Concurrent executions on one session corrupt transcript: no concurrency guard, and pre-turn repair runs against in-flight turns

[danshapiro]

Summary

A session becomes permanently unresumable with Anthropic 400 unexpected tool_use_id found in tool_result blocks when:

1. A turn parks on asyncio.gather in a parallel tool group (one tool finishes quickly, another runs 9+ hours)
2. A second execution starts on the same root session while the first is parked
3. Pre-turn repair (main.py:2756) synthesizes placeholder tool_results and repairs the transcript WITHOUT coordinating with the in-flight turn
4. When the parked turn's gather returns, it appends real tool_results to the in-memory context AFTER the new turn's assistant message
5. The next provider request sees tool_results with no matching tool_use in the immediately preceding message → 400

The session is thereafter permanently corrupted on every subsequent request until manually repaired.

Root Cause: Three Defects

Defect A — No concurrency guard on execute()
amplifier_app_cli/main.py:2871–2882 — the async executor assumes sequential execution within a session. Two concurrent execute() coroutines can mutate the same shared context.messages list:

# main.py:2871–2882
async def execute(self, *, engine, provider, budget=None, turn_limit=None):
    context = self.context  # shared reference
    # ... no guard prevents a second execute() on same session from running here ...

Defect B — Pre-turn repair mutates state owned by in-flight execution
amplifier_app_cli/main.py:2756 calls _repair_transcript_if_needed() at turn start WITHOUT cancelling or coordinating with an ongoing, parked execution:

# main.py:2756
await session.message_loop._repair_transcript_if_needed()  # diagnoses + mutates context

When an earlier asyncio.gather is still blocked waiting on a slow tool (9h+ delegate in the case that triggered this), _repair_transcript_if_needed() rewrites the transcript, injecting synthetic tool_results and assistant messages into a context that a parked coroutine will soon mutate further. No await, no cancel, no coordination.

Defect C — No validation at append sites that tool_use/tool_result pairing is maintained
Three append locations never check does_this_tool_call_id_already_have_a_result:

• amplifier_foundation/context/context_simple.py — add_message()
• amplifier_foundation/loop_streaming/__init__.py:849–857 — when gather() returns, each tool_result is appended
• amplifier_foundation/session/session_store.py:103 — when transcript is persisted

Any can silently append a duplicate or out-of-order result.

Proof: Event Log Timeline

Session 8cbc9006-9e2a-4be6-b1a4-151ed5c3575d (July 12–13, 2026):

2026-07-12 15:07:47 exec-B issue 2 tool_calls (todo + delegate)
2026-07-12 15:07:47 exec-B tool:post todo (5 ms) ← held in gather, todo result ready
                    delegate(agent=self) still running...
                    [9 hours 18 minutes elapse]
2026-07-13 00:25:32 exec-C prompt:submit + execution:start (ROOT SESSION)
2026-07-13 00:25:32 exec-C llm:request (concurrent with exec-B still parked)
2026-07-13 00:25:50 exec-C tool:pre/post bash
2026-07-13 00:25:52 exec-B tool:post delegate ← finally returns after 9h18m
2026-07-13 00:25:52 exec-B llm:request ← tries to send context to Anthropic
2026-07-13 00:25:53 exec-B provider:error 400 "unexpected tool_use_id..."
2026-07-13 00:25:53 exec-B orchestrator:error

Execution B and Execution C were concurrent in the same process on the same session's context. No execution lifecycle event shows cancellation or coordination. The events.jsonl confirms overlapping llm:request from both, proving concurrent mutation of shared context.

What Happened in the Transcript

Pre-turn repair running at exec-C:00:25:32 saw line 116 (the unresolved tool_calls from exec-B) and injected synthetic placeholder results at lines 117–118, then an auto-repaired assistant message (line 121). This went into the in-memory context.

When exec-B's gather() returned 20 seconds later (line 123–124 timestamps), it appended the real todo and delegate results into the context list. But they landed after line 121's assistant message, creating this structure:

Line 116: assistant [tool_call todo, tool_call delegate]  ← exec-B
Line 117–118: synthetic placeholders [no metadata] (from repair)
Line 119–120: user message
Line 121: assistant (from pre-turn repair)  ← exec-C turn
Line 122: tool (bash result)
Line 123–124: REAL results [with metadata] ← exec-B's gather() returned here

Anthropic rejects this because line 123–124 have tool_use_id that point back to line 116, but line 121's assistant message (which Anthropic assumes owns them) has no such tool_call.

Consequences

• Session permanently unresumable with 400 on every turn until manually repaired
• No automatic fix available: diagnosis.py sees the transcript "healthy" (see Issue 2)
• User must manually delete lines 123–124 or edit transcript.jsonl directly
• Self-driving / automation harnesses are completely blocked (cannot resume)
• Affected session evidence (events.jsonl, original transcript.jsonl.backup) available on request; timestamps and full event sequence included above

Environment

• amplifier-app-cli 0.1.1
• Anthropic provider
• bundle: self-driving
• Linux / WSL2
• Session triggered by interactive execution + concurrent self-driving agent harness orchestration

Honest Caveat

What launched exec-C (the second execution on the live session) is not identified from the evidence. Interactive REPL is strictly sequential; the log shows a self-driving automation harness was driving the session. No external caller should be able to corrupt a session by launching a second execution, but the root cause chain (overlapping executions on one context) is proven by event sequence and code flow.

Proposed Fix Direction

1. Execute-level concurrency guard: Serialize execute() calls per session, or add a hard gate that rejects a second execution until the first is orchestrator:complete and the context is drained.
2. Pre-turn repair coordination: If repair detects in-flight execution (a parked gather task or a non-complete orchestrator), either cancel it with a synthetic error or skip repair for this turn.
3. Append-site validation: At every add_message() append or tool_result append, validate tool_use/tool_result pairing — if a tool_result already exists for this ID, error or merge, never silently duplicate.

Related: Issue [ISSUE-2-URL] (diagnosis.py blind spots that keep the corrupt session undetectable)