[provider-openai] Response chaining sends previous_response_id AND full conversation history, double-counting server-side context

[danshapiro]

Summary

With enable_response_chaining: auto (the default behavior when the model supports reasoning), the OpenAI provider sends both previous_response_id and the full re-converted conversation history as input on each request. Server-side, the chained prior context is then counted in addition to the re-sent history, inflating effective context usage and accelerating context_length_exceeded failures (see #320 for the downstream failure mode).

Environment

• Amplifier CLI: 2026.07.11-c150034
• amplifier-core: 1.6.0
• Provider: OpenAI (gpt-5.6-sol), enable_response_chaining: auto, enable_long_context: true

Details

In amplifier_module_provider_openai/__init__.py (current module cache):

• At ~:1050-1053 and ~:1091-1092, when a chained response id is available the request still includes the full converted message history in input alongside previous_response_id.
• The code itself acknowledges the double-count hazard for the continuation path at ~:1860-1864.

The azure-openai provider reuses OpenAIProvider and inherits the behavior.

Expected behavior

When previous_response_id is sent, input should contain only the delta (new messages since the chained response), or chaining should be disabled and full history sent — not both.

Actual behavior

Both are sent; server-side context consumption roughly doubles relative to the visible conversation, so overflow occurs well before the local token estimate predicts.

Suggested fix

• When chaining is active, send only messages added since the last chained response.
• Add a regression test asserting that a chained request's input excludes messages already covered by previous_response_id.

Related

• Large tool results bypass auto-compaction and context_length_exceeded is retried unchanged #320 (compaction bypass + retry of context_length_exceeded)

[danshapiro]

Confirmed compaction-specific reproduction

A sanitized analysis of a long-running incremental/full-history session on provider-openai / gpt-5.6-sol confirms this issue and shows why local compaction does not recover the session:

• Local compaction did work on the local transcript: one representative compaction reduced its estimate from ~474k to ~129–132k tokens and left roughly 110 messages. Across the run, 17 compactions kept the local message count around 105–122.
• Response chaining remained active across those compactions: previous_response_id was present in 158 of 168 provider requests.
• Despite the locally compacted transcript, provider-reported usage.input_tokens grew monotonically:
  97,445 -> 190,476 -> 419,902 -> 611,110 -> 792,720 -> 852,452 -> 911,474
  (cache_read=0 for these observations).
• gpt-5.6-sol successfully accepted 911,474 input tokens before the next turn overflowed. The observed 203,104 value was Amplifier's local compaction budget, not the model context-window limit.
• Once the accumulated Responses-API chain exceeded the effective model limit, Amplifier performed two rounds of five context_length_exceeded retries without breaking the chain, so each retry repeated the same deterministic failure.

Verified root cause

Compaction reduces Amplifier's local transcript, but provider-openai continues sending previous_response_id. OpenAI therefore reconstructs the accumulated server-side Responses-API chain, so the effective provider input keeps growing even while the local transcript stays compact. This is the response-chaining defect described here, not primarily a local compaction-target/headroom failure.

Compaction-specific fix requirements

In addition to avoiding full-history-plus-chain double counting generally:

1. When compaction occurs, clear/reset previous_response_id and submit the compacted transcript as a fresh chain.
2. Incorporate provider-reported usage into compaction/chaining decisions; a local transcript estimate alone cannot represent retained server-side chain usage.
3. On context_length_exceeded, break the chain and retry safely with the compacted transcript rather than retrying the same oversized chain.
4. Add a regression test in which local message count shrinks after compaction while provider input usage would otherwise continue growing through previous_response_id.

No raw transcript, local paths, secrets, or personal content are included. Detailed local event metadata is available if maintainers need it. This evidence supersedes the initial root-cause theory in #328, which is being closed as a duplicate of this issue.

[danshapiro]

Follow-up: response chain persists across full process restart

Evidence from session restart: 507f3164-6b1e-48ff-a8ce-766e5cdbde76

After the user killed the stuck process and did a genuine full restart (confirmed via session:resume lifecycle event), the session appeared to recover locally but failed again immediately on the first LLM request.

Timeline:

• Last activity before restart: 2026-07-18T13:20:36Z
• Session resume event: 2026-07-18T14:03:10Z (40+ minute gap)
• Transcript grew: 112 → 119 messages (fresh local content loaded)
• First LLM request after restart: still FAILED with context_length_exceeded

Root cause: previous_response_id chain was NOT reset across restart

The exact same OpenAI response_id (resp_0d835b02e4b92619006a5b7b9135a88198a7f10d4f934b43a3, originally created at 2026-07-18T13:12:20Z) appeared in both:

• Last llm:request before crash (13:18:53Z)
• First llm:request after restart (14:03:10Z)

This value is not stored in the session's on-disk metadata.json, indicating it's either embedded in provider-specific state OR reconstructed by the orchestrator from turn history during resume. Critically: there is no explicit "clear previous_response_id on session:resume" logic in provider-openai.

Implication: Killing and restarting the process does NOT break the OpenAI response chain. The bug is not fixed by retry/restart — it requires a code fix in provider-openai to explicitly reset previous_response_id when a new process starts or a session:resume occurs, ensuring OpenAI doesn't try to reconstruct the pre-compaction context server-side.

Sanitization note: Timestamps, event types, message counts, and response_id (opaque OpenAI identifier) only — no file paths, transcript content, or secrets.

[danshapiro]

Correction + confirmed root cause: chain pointer is persisted on disk in the transcript

This supersedes the uncertainty in my previous comment (which speculated the id was "not stored on disk ... either embedded OR reconstructed"). Verified at the code + data level: the chain pointer is persisted on disk, in the transcript, and provider-openai rebuilds the chain from it on resume. Same session: 507f3164-..., gpt-5.6-sol via provider-openai.

What actually happens on resume

• A genuine full process restart occurred: session:resume at 2026-07-18T14:03:10Z (new PID; confirmed real reload, not a retry).
• The first LLM request after restart failed at 2026-07-18T14:04:50Z with context_length_exceeded, and it reused the exact same previous_response_id as the last pre-restart success (resp_0d835b02..., originally created 13:12:20Z). Restart did not break the chain.

Why restart cannot clear it (persisted, then reverse-scanned)

1. Written to disk every assistant turn. The provider stores the server response id into assistant-message metadata under key openai:response_id (_response_handling.py, metadata[METADATA_RESPONSE_ID] = final_response.id). This metadata is serialized into transcript.jsonl. In this session, every one of the 161 assistant messages carries openai:response_id.
2. Re-lifted on every request, including the first after resume. On request build, provider-openai reverse-scans the loaded message list for the most recent assistant message and lifts its stored id back into previous_response_id (__init__.py, reverse scan for last assistant message). So resume re-reads the pre-compaction chain pointer straight off disk and hands it back to OpenAI, which reconstructs the full pre-compaction server-side context.

This is why "kill and re-resume" reproduced the identical failure: restart clears in-memory state but not the on-disk chain pointer.

Why a one-turn rewind is not a fix

Trimming the last turn does not help — the preceding assistant messages still carry their own openai:response_id values, and those chains still represent the bloated (~852k-token) server-side input. Reverse-scan simply picks up the next assistant id and rebuilds the same oversized chain.

Recovery + fix

• Supported workaround (recovery): set provider config enable_response_chaining=false for this session/provider. This stops previous_response_id from being sent, so OpenAI stops reconstructing the pre-compaction chain and the compacted transcript is sent as the request instead.
• Durable fix: drop the response chain (do not send previous_response_id, and/or reset the persisted pointer) whenever the context is reset — specifically after compaction and on context_length_exceeded — so the next request starts a fresh chain from the compacted transcript.

Transcript integrity is healthy (no orphaned/misordered tool results) — this is purely the response-chain persistence/reconstruction defect, not corruption. Do not rewind.

Sanitization: event types, timestamps, message counts, opaque OpenAI response id, and provider-module source-file references only — no local filesystem paths, transcript content, or secrets.

[danshapiro]

⚠️ Workaround Status Update: Session-Scoped Settings Override Does Not Work

Session: 507f3164-6b1e-48ff-a8ce-766e5cdbde76

The recommended workaround in this issue — using a session-scoped settings.yaml with enable_response_chaining: false — does not currently work due to a separate defect in amplifier-app-cli.

The Issue

amplifier resume <session_id> silently ignores session-scoped settings.yaml provider overrides. The resume code path builds an unscoped AppSettings() instead of AppSettings().with_session(session_id, project_slug) at commands/session.py:131, so the session-scoped settings file is never read.

Why We Know It's Not a User Error

• The override file was correctly written to the right location
• An offline check using .with_session() directly confirmed the override would resolve to False
• Live amplifier resume immediately failed with the identical error message

This proves the settings mechanism itself works, but the resume code path doesn't load it.

Tracking

A detailed bug report with full evidence and suggested fix has been filed at #329 (in the core repo, since amplifier-app-cli has issues disabled).

Durable Recovery Path

Until that defect is fixed:

1. Use the response-chaining fix being delivered in provider-openai (Option A + B from amplifier-support#208)
2. Do NOT rely on session-scoped settings overrides for recovery — they are non-functional on resume
3. Apply provider config changes to your global/project-level settings.yaml and start a fresh session instead