Skip to content

Fix silent corruption for large backreference offsets#22

Open
erseco wants to merge 2 commits into
101arrowz:masterfrom
erseco:fix-large-backreference-offsets
Open

Fix silent corruption for large backreference offsets#22
erseco wants to merge 2 commits into
101arrowz:masterfrom
erseco:fix-large-backreference-offsets

Conversation

@erseco

@erseco erseco commented Jul 4, 2026

Copy link
Copy Markdown

Related to #19

Summary

fzstd could silently decode the wrong bytes when a sequence's backreference offset needed more bits than the fixed four-byte read made available after alignment.

For sh = spos & 7, the existing read exposes only 32 - sh usable bits. Offset codes ofc >= 26 can therefore cross into a fifth byte when sh + ofc > 32. The first affected match distance is approximately 96 MiB (2^26 + 2^25 - 3), so this is relevant to frames that actually use long-distance matches, typically with windowLog >= 27, --long, or suitable ultra-level settings.

The leading 2^ofc term remained intact, so the truncated value was normally still in range. The decoder then copied from a nearer, incorrect position without throwing, while preserving the output length.

Fix

  • Read the fifth byte only when the offset field crosses the 32-bit reading window.
  • Reject offset codes above 30, which cannot be represented safely within the decoder's accepted approximately 2 GiB window and signed repeat-offset storage.
  • Reuse the bit alignment and mask within the sequence loop.
  • Narrow the README wording: offset decoding now covers the accepted frame window, but very large frames remain subject to JavaScript typed-array, memory, and frame-size limits.

Validation

The offset arithmetic was checked against a BigInt reference for every ofc from 0 through 30, all eight bit alignments, and randomized five-byte inputs.

The included regressions cover:

  • one-shot and streaming decoding;
  • every two-chunk split for the embedded frames;
  • one-byte pushes and never-final mode;
  • concatenated frames;
  • an opt-in zstd -1 --long=27 corpus with approximately 100 MiB match distances.

Local large-test result:

  • input: 106,857,600 bytes;
  • compressed: 104,862,815 bytes;
  • output: byte-identical;
  • peak RSS with Node.js 22: approximately 758 MiB; the test documentation rounds this to 800 MB.

Performance

The hot sequence path gains one range check and one predictable conditional. The fifth-byte read is only executed for offset fields that actually cross the four-byte window. No end-to-end performance claim is made by this PR.

Scope

This is separate from upstream PR #21, which fixes int32 overflow in bit positions for one-shot frames whose compressed size exceeds 256 MiB. The changes are compatible.

The historical "hello world ".repeat(11_000) streaming symptom occurs on fzstd <= 0.1.0 and is already fixed on current master; the new small regression cases keep that behavior covered.

The 4-byte offset bit-read in the sequence loop supplies only
32 - (spos & 7) usable bits, so offset codes >= 26 could silently lose
their high bits whenever the field straddled past that window. The
2^ofc leading term survives, leaving an in-bounds but too-small offset
that splices data from the wrong position: output length stays correct,
no error is thrown, and decompress() and Decompress corrupt
identically. Read the fifth byte when the field needs it.

Also adds streaming regression tests for the issue 101arrowz#19 scenario
(embedded frames, exhaustive chunk-split sweeps, never-final mode,
multi-frame) plus an opt-in large-offset test (FZSTD_BIG_TESTS=1), and
updates the README note about the former 2^25 backreference limit.
Reject offset codes outside the decoder's signed 31-bit range, reuse the bit alignment and mask in the sequence loop, and keep the fifth-byte read limited to fields that actually cross the 32-bit window.

Also narrow the README claims to the support actually provided and correct the large regression test's threshold, generator name, and measured memory requirement.
@erseco erseco force-pushed the fix-large-backreference-offsets branch from 4d6eb36 to 15007f3 Compare July 11, 2026 11:57
@erseco erseco changed the title Fix silent corruption for backreference offsets beyond 2^25 Fix silent corruption for large backreference offsets Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant