Skip to content

feat(llm, config): Cap responses at a configurable byte ceiling - #923

Merged
JeanMertz merged 6 commits into
mainfrom
opus-bat-sht-crazy
Jul 31, 2026
Merged

feat(llm, config): Cap responses at a configurable byte ceiling#923
JeanMertz merged 6 commits into
mainfrom
opus-bat-sht-crazy

Conversation

@JeanMertz

@JeanMertz JeanMertz commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

A query now aborts once a response's generated content passes assistant.request.max_response_bytes (default 1 MiB, 0 disables it). The turn ends with an error, but any content already streamed stays in the conversation, and the response is not retried, since retrying would just regenerate the same runaway. This bounds the cost of a model that gets stuck generating without end while nobody is watching the terminal.

The ceiling also applies to the requests that run with no terminal attached: title generation, conversation summarize, conversation edit title generation, and tool inquiries. Those collect a whole response before using it, so a breach there fails outright and keeps no partial result. They take only the ceiling from assistant.request, not max_retries or the backoff values, which stay at their own defaults: the retry settings are tuned for a query the user is watching, and an unattended background request that inherited a large retry budget would stay alive far longer than its caller expects.

The ceiling wraps the stream outside any provider-level continuation logic, so the bytes of every chained continuation accumulate against a single ceiling rather than resetting per link. It counts the bytes JP receives, which can be fewer than the bytes billed: Anthropic's continuation chaining drops the reasoning of each continuation and trims the overlap between links before yielding events, so those bytes are paid for but never counted. The ceiling therefore bounds retained output rather than spend. The number of continuation requests is bounded separately: Anthropic's chaining previously recursed without limit, and is now capped at 5 links (MAX_CHAIN_DEPTH), so a model that keeps hitting the token ceiling can no longer chain requests forever.

The count covers assistant messages, reasoning, structured output, tool-call arguments, and the id and name of each tool call. Flushes, patches, and keep-alives carry no generated content and do not count.

Bytes are counted rather than tokens, since tokens can't be counted locally; roughly four bytes per token is a reasonable guide when tuning the value. Set it from the CLI, e.g.
jp config set assistant.request.max_response_bytes 4096.

A query now aborts once a response's generated content passes
`assistant.request.max_response_bytes` (default 1 MiB, `0` disables
it). The turn ends with an error, but any content already streamed
stays in the conversation, and the response is not retried, since
retrying would just regenerate the same runaway. This bounds the cost
of a model that gets stuck generating without end while nobody is
watching the terminal, and applies equally to interactive queries and
to no-terminal collect calls such as title generation and
summarization.

The ceiling wraps the stream outside any provider-level continuation
logic, so a response Anthropic assembles from several chained
`MaxTokens` continuations counts as one response against the limit
rather than resetting per link. Anthropic's own continuation chaining
is separately bounded to 5 links (`MAX_CHAIN_DEPTH`), so a model that
keeps hitting the token ceiling can no longer chain requests forever.

Bytes are counted rather than tokens, since tokens can't be counted
locally; roughly four bytes per token is a reasonable guide when
tuning the value. Set it from the CLI, e.g.
`jp config set assistant.request.max_response_bytes 4096`.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
@JeanMertz
JeanMertz force-pushed the opus-bat-sht-crazy branch from b55d6b8 to 704f2d9 Compare July 31, 2026 13:33
Signed-off-by: Jean Mertz <git@jeanmertz.com>
@JeanMertz
JeanMertz merged commit bf03db6 into main Jul 31, 2026
16 checks passed
@JeanMertz
JeanMertz deleted the opus-bat-sht-crazy branch July 31, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant