Skip to content

Reading an .ndx cache walks the parser past the end of its buffer#811

Open
xroche wants to merge 4 commits into
masterfrom
fix/ndx-full-length-fields-793
Open

Reading an .ndx cache walks the parser past the end of its buffer#811
xroche wants to merge 4 commits into
masterfrom
fix/ndx-full-length-fields-793

Conversation

@xroche

@xroche xroche commented Jul 27, 2026

Copy link
Copy Markdown
Owner

binput() returns count + 1, assuming it consumed a separator. It did not when it stopped on the buffer's terminating NUL, or because the destination filled: that byte is data, and the caller's cursor ends up one past it. In PT_LoadCache__Old the cursor then sits outside the heap buffer holding the .ndx, and the next binput() call reads out of bounds. It is a read and not a write. The trigger is the walk reaching use[ndxSize], which any index that does not end in a newline reaches, so full-length URL fields are one route into it and not the only one; a field a byte under capacity that ends at EOF gets there too. Only the legacy .ndx reader is affected, and everything HTTrack writes ends in a newline, so it takes a hand-made or corrupt index. The loop's a < (use + ndxSize) guard cannot catch any of this, because the drift happens after the check.

Step over the byte only when it really is a newline. cache_brstr() had to follow, since it carried a caller-side patch for the same drift and its adr[off - 1] probe would read before the buffer now that binput() can return 0. Test 92 covers the neighbouring write overflow from #744 and appends a trailing entry on purpose, so the drifted cursor still lands inside the malloc'd buffer and 92 passes on the unfixed tree. Test 117 drives all three field bounds (1024, whatever the first URL half leaves, 200), cache_brstr's own 256, and a field that fills none of them. Two of its cases have teeth outside the sanitizer build: two URLs differing only on the 1024 bound must stay two cache keys, and a header one byte past 256 must parse like a header two bytes past. On master each loses an entry.

Closes #793

binput() returned count + 1 unconditionally, assuming it had consumed a
separator. When it stopped on the buffer's terminating NUL, or because the
destination filled, that byte was not a separator and the caller's cursor
moved one past it. In PT_LoadCache__Old the cursor then leaves the heap
buffer holding the .ndx and the next binput() call reads out of bounds; the
loop's a < (use + ndxSize) guard cannot see it, since the drift happens
after the check.

Step over the byte only when it really is a newline. cache_brstr() carried
a caller-side patch for the same drift and had to follow, or its adr[off-1]
probe would read before the buffer once binput() can return 0.

Closes #793

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
xroche and others added 2 commits July 27, 2026 14:26
The trigger is the walk reaching the buffer's terminating NUL, which any
.ndx not ending in a newline does; full-length fields are one route to it,
not the only one. Add a field that stops short of every bound, and a pair
straddling cache_brstr's own 256-byte field, the second destination the
diff touches.

items() returned 0 whenever the item-count line was missing, so a crawl
that printed nothing passed the zero-key assertion. Signals now clean up on
their own trap line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
@xroche xroche changed the title Reading an .ndx cache with full-length URL fields reads past the buffer Reading an .ndx cache walks the parser past the end of its buffer Jul 27, 2026
Windows resolves NUL.<anything> to the null device, so nul.ndx never
existed there and proxytrack reported an unloadable index. The suite's
Win32 leg caught it once items() stopped treating a missing count as zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reading an .ndx cache with full-length URL fields reads past the buffer

1 participant