Skip to content

Fix memory leaks in compression scratch buffers on exception - #139

Open
alexey-milovidov wants to merge 3 commits into
szcompressor:masterfrom
alexey-milovidov:fix-compress-scratch-buffer-leaks
Open

Fix memory leaks in compression scratch buffers on exception#139
alexey-milovidov wants to merge 3 commits into
szcompressor:masterfrom
alexey-milovidov:fix-compress-scratch-buffer-leaks

Conversation

@alexey-milovidov

Copy link
Copy Markdown

What this fixes

Lossless_zstd::compress throws std::length_error when the destination capacity is too small for poorly-compressible data, and the Huffman encoder can also throw during encode. Every compress-side scratch buffer in the library is a raw malloc with a bare free at the end of the function, so each failed compression leaks the buffer. A long-running application that catches the exception and keeps going (as ClickHouse does with its experimental SZ3 codec) accumulates the leaked buffers.

This was observed by LeakSanitizer in a ClickHouse stress test: 383 KB leaked in 2 allocations from SZ3::SZGenericCompressor::compress after compressions of poorly-compressible data failed with std::length_error.

The fix

Apply the same unique_ptr-with-free RAII pattern that SZGenericCompressor::decompress already uses to all compress-side scratch buffers:

  • SZGenericCompressor::compress
  • SZTruncateCompressor::compress
  • SZExaaltCompressor::compress (and its decompress, which parses untrusted data and could leak on corrupted input)
  • SZ_compress_dispatcher (the direct-zstd fallback buffer)
  • SZ_compress_Interp_lorenzo, interp_compress_test, lorenzo_compress_test

Additionally:

  • The heap-owning overload SZ_compress(const Config &, const T *, size_t &) in include/SZ3/api/sz.hpp allocated its output buffer with a bare new char[] and leaked it when the inner compression threw before the pointer reached the caller. It now holds the buffer in a std::unique_ptr and releases it only on success.
  • SZ_compress_OMP in include/SZ3/api/impl/SZImplOMP.hpp kept each thread's scratch chunk in a raw malloc buffer freed only after SZ_compress_dispatcher returned, so a chunk that threw leaked the buffer. Moreover the per-chunk capacity ZSTD_compressBound(num * sizeof(T)) did not reserve room for the size_t header that Lossless_zstd::compress writes in front of the zstd stream, so the direct lossless path threw for poorly-compressible chunks. Both are fixed, and SZ_compress_size_bound_omp reserves the header space accordingly.

Behavior

No behavior change for successful compressions; failures no longer leak.

Testing

  • Built with -DBUILD_TESTING=ON; all 7 ctest tests pass.
  • Round-trip smoke test with the sz3 CLI on random data (-3 10 20 30 -M ABS 1e-3 -a) stays within the error bound.
  • The _OPENMP-guarded path is not compiled by the default build, so it was verified by compiling the header standalone with g++ -std=c++17 -fopenmp -Wall -Wextra -fsyntax-only and explicit instantiation of SZ_compress_OMP<float, 3> and SZ_compress_size_bound_omp<float>: no errors.
  • The fix has been running in ClickHouse's fork of SZ3: Fix memory leak in SZ3 compression scratch buffers ClickHouse/ClickHouse#113916

Lossless_zstd::compress throws std::length_error when the destination
capacity is too small for poorly-compressible data, and the Huffman
encoder can throw on encode. Every compress-side scratch buffer was a
raw malloc with a bare free() at the end of the function, so each
failed compression leaked the buffer (observed by LeakSanitizer in
ClickHouse stress tests via SZGenericCompressor::compress). Own the
buffers with the same unique_ptr-with-free RAII pattern already used
in SZGenericCompressor::decompress, and apply it to the Exaalt
decompression path too.
The overload SZ_compress(const Config &, const T *, size_t &) allocated its
output buffer with a bare new char[] and returned it raw; if the inner
compression (e.g. Lossless_zstd::compress on poorly-compressible data) threw,
the buffer leaked. Hold it in std::unique_ptr and release only on success.
SZ_compress_OMP kept each thread's scratch chunk in a raw malloc buffer and
only released it after SZ_compress_dispatcher returned, so a chunk that threw
(for example std::length_error from Lossless_zstd::compress) leaked the buffer.
Own the buffer with std::unique_ptr and reserve room for the size header that
Lossless_zstd::compress writes in front of the zstd stream, so the direct
lossless path does not throw for poorly compressible chunks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant