Fix undefined behavior casting non-finite / out-of-range float to int in LinearQuantizer - #138
Open
alexey-milovidov wants to merge 1 commit into
Open
Conversation
…verflow LinearQuantizer::quantize_and_overwrite cast fabs(diff) * error_bound_reciprocal directly to int64_t. When the input value is NaN, diff is NaN; for infinities or huge magnitudes the product exceeds the int64_t range. Converting such a value to an integer is undefined behaviour, reported by UBSan as '... is outside the range of representable values of type long'. Check the finite quantization range on the double before the cast. Values that cannot be represented as an index (NaN, +/-Inf, overflow) are stored losslessly in unpred, which is the same outcome the out-of-range branch already produced for finite values, so quantization of representable values is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
LinearQuantizer::quantize_and_overwritecomputes a quantization index by casting a floating-point product toint64_t:When the input
dataisNaN,diffisNaN; for infinities or very large magnitudes the product exceeds theint64_trange. Converting such a value to an integer is undefined behavior in C++. It is reported by UndefinedBehaviorSanitizer underfloat-cast-overflowas... is outside the range of representable values of type 'long', and yields a garbage index on any platform.Fix
Compute the scaled magnitude as a
doubleand check that it is within the representable quantization range before the cast.NaNfails thescaled < radius * 2comparison (any comparison withNaNis false) and infinities/overflows fail it too, so every non-representable value falls through to be stored losslessly inunpred— exactly the same outcome the existing out-of-range branch already produced for finite values. Quantization of representable values is unchanged.Context
Found while integrating SZ3 into ClickHouse (UBSan build, compressing float columns that contain
NaN/Inf). Companion fix on the ClickHouse fork: ClickHouse#3