fix: fail fast on client send when payload exceeds MAX_TOTAL_CHUNKS#81
Merged
Conversation
The WebSocket client's send path chunked a large ClientRequest and streamed every chunk without checking MAX_TOTAL_CHUNKS. When a payload exceeds the cap (256 chunks x 256 KiB = 64 MiB), the client streamed the ENTIRE oversized payload to the node, which rejects the first chunk during reassembly (ReassemblyBuffer::receive_chunk returns TotalChunksTooLarge). Every byte after the first chunk was wasted bandwidth (a ~450 MiB payload would upload in full before the node rejects it). Add a reusable, tested ensure_chunkable(len) helper in streaming.rs that mirrors the node's reassembly cap, and call it on both send paths (regular/native and browser/wasm) before any chunk is sent, so an oversized request fails fast with a clear TotalChunksTooLarge error and nothing is streamed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a call-site regression test (test_send_oversized_rejected_before_streaming) that drives a real WebApi over a localhost socket with an oversized Put (state one byte over the 64 MiB MAX_TOTAL_CHUNKS * CHUNK_SIZE cap) and asserts recv() returns a "exceeds maximum" (TotalChunksTooLarge) error. Verified that removing the ensure_chunkable(...) guard from process_request makes this test fail (the client streams the payload and recv() instead returns a connection-reset error), so the guard can't be silently deleted without CI catching it. The prior ensure_chunkable unit tests only covered the helper, not the wiring. Also document at the guard call site that returning Err tears down the WebApi connection (request_handler break), which is intentional and acceptable for an unsendable over-cap request: the error is still delivered to recv() first, and the browser/wasm path surfaces it without teardown. We deliberately do not thread response_tx through process_request for per-request error reporting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The WebSocket client's send path chunks a large
ClientRequestand sends every chunk without checkingMAX_TOTAL_CHUNKS. The cap is 256 chunks x 256 KiB = 64 MiB (MAX_TOTAL_CHUNKS * CHUNK_SIZE).When a serialized request exceeds that cap, the client streams the entire oversized payload to the node, which then rejects the very first chunk during reassembly (
ReassemblyBuffer::receive_chunkreturnsTotalChunksTooLargefortotal > MAX_TOTAL_CHUNKS). Every byte after the first chunk is wasted: a payload well above the 64 MiB cap is uploaded in full only to be discarded on arrival.Both send paths had this gap:
regular.rs::process_request(native/tokio path used by fdev)browser.rs::send(wasm path used by River)Solution
Add a small, reusable, tested helper in
streaming.rs:It computes the chunk count with the same math as
chunk_bytes(len.div_ceil(CHUNK_SIZE).max(1)) so the client'stotalequals what the node validates on the wire, and returnsStreamError::TotalChunksTooLarge { total, max }when it would exceedMAX_TOTAL_CHUNKS.Both send paths call
ensure_chunkable(msg.len())before entering the chunking branch, so an oversized request fails fast with a clear error and no chunk is sent:regular.rs: maps the error intoError::OtherErrorand returns it; the request handler surfaces it to the caller (the client'srecv()gets the error).browser.rs: surfaces it via the existingError::ConnectionError(serde_json::Value)+error_handlerpattern, matching how the file already reports send failures to the JS caller.The guard mirrors the node's own reassembly cap — this is a client-side pre-check of an invariant the node already enforces.
Note on error propagation (
regular.rs)Returning
Errfromprocess_requestbreaks the request-handler loop, which tears down theWebApiconnection after delivering the error torecv(). For an unsendable request that is acceptable behavior. Surfacing the error for just this request without tearing down the connection would require threadingresponse_txintoprocess_request(a signature/flow change); it was not deemed worth the added complexity for a request that cannot be sent regardless. The caller still receives the clearTotalChunksTooLargeerror either way.Testing
Added three cheap, deterministic unit tests in
streaming.rs(they pass only the length integer — no multi-MiB payload is allocated):ensure_chunkable_rejects_oversized—(MAX_TOTAL_CHUNKS + 1) * CHUNK_SIZEis rejected withTotalChunksTooLarge { total: 257, max: 256 }.ensure_chunkable_accepts_at_limit— exactlyMAX_TOTAL_CHUNKS * CHUNK_SIZEis accepted; one byte over is rejected.ensure_chunkable_accepts_small— 0, 1 KiB, andCHUNK_THRESHOLDare accepted.No dedicated
regular.rssend-path test: triggering the guard end-to-end requires a >64 MiB serialized request, i.e. a >64 MiB allocation, which these tests deliberately avoid. The helper unit tests plus the existing node-sidetotal_too_large_errorreassembly test cover the behavior.Local verification:
cargo fmt— cleancargo clippy --all-targets --features net -- -D warnings— cleancargo test --workspace --features contract,net,testing,trace(the CI test command) — all pass, incl. the 3 new testscargo build --target wasm32-unknown-unknown --features net --lib— compiles (CI's wasm jobs run on default features and don't compile thenet-gatedbrowser.rs, sobrowser.rswas verified locally against the wasm target)Companion to freenet/freenet-core#4653
[AI-assisted - Claude]