Fix memory leaks in compression scratch buffers on exception - #139
Open
alexey-milovidov wants to merge 3 commits into
Open
Fix memory leaks in compression scratch buffers on exception#139alexey-milovidov wants to merge 3 commits into
alexey-milovidov wants to merge 3 commits into
Conversation
Lossless_zstd::compress throws std::length_error when the destination capacity is too small for poorly-compressible data, and the Huffman encoder can throw on encode. Every compress-side scratch buffer was a raw malloc with a bare free() at the end of the function, so each failed compression leaked the buffer (observed by LeakSanitizer in ClickHouse stress tests via SZGenericCompressor::compress). Own the buffers with the same unique_ptr-with-free RAII pattern already used in SZGenericCompressor::decompress, and apply it to the Exaalt decompression path too.
The overload SZ_compress(const Config &, const T *, size_t &) allocated its output buffer with a bare new char[] and returned it raw; if the inner compression (e.g. Lossless_zstd::compress on poorly-compressible data) threw, the buffer leaked. Hold it in std::unique_ptr and release only on success.
SZ_compress_OMP kept each thread's scratch chunk in a raw malloc buffer and only released it after SZ_compress_dispatcher returned, so a chunk that threw (for example std::length_error from Lossless_zstd::compress) leaked the buffer. Own the buffer with std::unique_ptr and reserve room for the size header that Lossless_zstd::compress writes in front of the zstd stream, so the direct lossless path does not throw for poorly compressible chunks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Lossless_zstd::compressthrowsstd::length_errorwhen the destination capacity is too small for poorly-compressible data, and the Huffman encoder can also throw during encode. Every compress-side scratch buffer in the library is a rawmallocwith a barefreeat the end of the function, so each failed compression leaks the buffer. A long-running application that catches the exception and keeps going (as ClickHouse does with its experimental SZ3 codec) accumulates the leaked buffers.This was observed by LeakSanitizer in a ClickHouse stress test: 383 KB leaked in 2 allocations from
SZ3::SZGenericCompressor::compressafter compressions of poorly-compressible data failed withstd::length_error.The fix
Apply the same
unique_ptr-with-freeRAII pattern thatSZGenericCompressor::decompressalready uses to all compress-side scratch buffers:SZGenericCompressor::compressSZTruncateCompressor::compressSZExaaltCompressor::compress(and itsdecompress, which parses untrusted data and could leak on corrupted input)SZ_compress_dispatcher(the direct-zstd fallback buffer)SZ_compress_Interp_lorenzo,interp_compress_test,lorenzo_compress_testAdditionally:
SZ_compress(const Config &, const T *, size_t &)ininclude/SZ3/api/sz.hppallocated its output buffer with a barenew char[]and leaked it when the inner compression threw before the pointer reached the caller. It now holds the buffer in astd::unique_ptrand releases it only on success.SZ_compress_OMPininclude/SZ3/api/impl/SZImplOMP.hppkept each thread's scratch chunk in a rawmallocbuffer freed only afterSZ_compress_dispatcherreturned, so a chunk that threw leaked the buffer. Moreover the per-chunk capacityZSTD_compressBound(num * sizeof(T))did not reserve room for thesize_theader thatLossless_zstd::compresswrites in front of the zstd stream, so the direct lossless path threw for poorly-compressible chunks. Both are fixed, andSZ_compress_size_bound_ompreserves the header space accordingly.Behavior
No behavior change for successful compressions; failures no longer leak.
Testing
-DBUILD_TESTING=ON; all 7 ctest tests pass.sz3CLI on random data (-3 10 20 30 -M ABS 1e-3 -a) stays within the error bound._OPENMP-guarded path is not compiled by the default build, so it was verified by compiling the header standalone withg++ -std=c++17 -fopenmp -Wall -Wextra -fsyntax-onlyand explicit instantiation ofSZ_compress_OMP<float, 3>andSZ_compress_size_bound_omp<float>: no errors.