feat(llm, config): Cap responses at a configurable byte ceiling - #923
Merged
Conversation
A query now aborts once a response's generated content passes `assistant.request.max_response_bytes` (default 1 MiB, `0` disables it). The turn ends with an error, but any content already streamed stays in the conversation, and the response is not retried, since retrying would just regenerate the same runaway. This bounds the cost of a model that gets stuck generating without end while nobody is watching the terminal, and applies equally to interactive queries and to no-terminal collect calls such as title generation and summarization. The ceiling wraps the stream outside any provider-level continuation logic, so a response Anthropic assembles from several chained `MaxTokens` continuations counts as one response against the limit rather than resetting per link. Anthropic's own continuation chaining is separately bounded to 5 links (`MAX_CHAIN_DEPTH`), so a model that keeps hitting the token ceiling can no longer chain requests forever. Bytes are counted rather than tokens, since tokens can't be counted locally; roughly four bytes per token is a reasonable guide when tuning the value. Set it from the CLI, e.g. `jp config set assistant.request.max_response_bytes 4096`. Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
JeanMertz
force-pushed
the
opus-bat-sht-crazy
branch
from
July 31, 2026 13:33
b55d6b8 to
704f2d9
Compare
Signed-off-by: Jean Mertz <git@jeanmertz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A query now aborts once a response's generated content passes
assistant.request.max_response_bytes(default 1 MiB,0disables it). The turn ends with an error, but any content already streamed stays in the conversation, and the response is not retried, since retrying would just regenerate the same runaway. This bounds the cost of a model that gets stuck generating without end while nobody is watching the terminal.The ceiling also applies to the requests that run with no terminal attached: title generation,
conversation summarize,conversation edittitle generation, and tool inquiries. Those collect a whole response before using it, so a breach there fails outright and keeps no partial result. They take only the ceiling fromassistant.request, notmax_retriesor the backoff values, which stay at their own defaults: the retry settings are tuned for a query the user is watching, and an unattended background request that inherited a large retry budget would stay alive far longer than its caller expects.The ceiling wraps the stream outside any provider-level continuation logic, so the bytes of every chained continuation accumulate against a single ceiling rather than resetting per link. It counts the bytes JP receives, which can be fewer than the bytes billed: Anthropic's continuation chaining drops the reasoning of each continuation and trims the overlap between links before yielding events, so those bytes are paid for but never counted. The ceiling therefore bounds retained output rather than spend. The number of continuation requests is bounded separately: Anthropic's chaining previously recursed without limit, and is now capped at 5 links (
MAX_CHAIN_DEPTH), so a model that keeps hitting the token ceiling can no longer chain requests forever.The count covers assistant messages, reasoning, structured output, tool-call arguments, and the id and name of each tool call. Flushes, patches, and keep-alives carry no generated content and do not count.
Bytes are counted rather than tokens, since tokens can't be counted locally; roughly four bytes per token is a reasonable guide when tuning the value. Set it from the CLI, e.g.
jp config set assistant.request.max_response_bytes 4096.