Notice when a Bot stops talking, instead of spinning forever

[jerelvelarde]

The problem

A Bot is any AG-UI endpoint, so it is infrastructure this deployment does not run: it gets redeployed mid-answer, its upstream times out, or it accepts a connection, writes a RUN_STARTED and never writes again. Nothing noticed. The stream stayed open, the channel stayed busy, the composer stayed locked, and the only way out was a reload or a restart.

Nor could anyone count it. One hung turn reads as a bad afternoon; the same Bot hanging twice a day for a month is a fact about an endpoint. The trail had no row for it: every event in audit.ts records something that happened, and this is the absence of anything happening.

The approach

Activity, not duration. A turn may run for an hour while events keep arriving; one whose stream has produced nothing at all for stallMs is wedged. The ceiling is on how long a person watches a spinner, not on how much work a Bot may do.

On the Bot's stream, not the reply to the browser. In Intelligence mode, OpenBot's only mode, POST /api/copilotkit/agent/:id/run does not stream: it answers fixed-length JSON in about a second, and events reach the browser over a WebSocket. The guard sits on the fetch an HttpAgent is built with; server/src/app.ts is untouched.

The clock is kept by the pump, not the relay. A TransformStream transforms only once the readable side is pulled, so timing there records when the consumer took a chunk, not when the Bot produced one — and that consumer is the Intelligence runner republishing over the network, which pauses. The relay is an identity transform, present only to give this process a writable end it can close from outside; the pump times each read off the Bot's body and stops the clock across the handover. A reader that stops for good is therefore never reported, which errs the safe way: believing a healthy Bot is broken ends somebody's turn. The relay may not buffer, await, or read the bytes.

What a stall does. Cancel the Bot's side, releasing the socket; queue one RUN_ERROR and close the stream; write the audit row last, since that store is a bare insert against the shared pool and a saturated pool is the condition a Bot is likeliest to hang in. A lost row beats a watchdog that fires and then parks. The sentence goes into every framing but the AG-UI protobuf media type, the rule @ag-ui/client 0.0.57 uses to pick its parser; being stricter would close a stream with nothing in it. Tool calls need no exemption: the run ends before the browser executes one.

Where the person reads it. Neither surface drew anything for a failed run, so both now draw the sentence from one source. The channel takes the shimmer's slot at the end of the transcript, and not as a message — no id, gone next turn — because the conversation goes back to the model. The direct Bot chat draws it under the page header, since the packaged chat virtualises its message list; the cost is that it is not beside the gap.

Configuration and the trail. AGENT_STALL_TIMEOUT_MS, read through config.ts, which refuses to boot on anything but a whole number of milliseconds; zero or unset switches the watch off, so no deployment acquires it unasked. .env.example ships 60000 because the deadline is in a race: Bots here serve on Bun with idleTimeout: 120, and Bun tears a wedged response down a second past that, handing the person a transport error and the trail no row. Each stall writes agent.stream_stalled with the Bot, thread, run, the length of the silence and how many chunks came first, which separates an endpoint that dies mid-answer from one that never answers. It joins the "Did not happen" filter and carries no actor.

What is not covered

• Built-in Bots. A BuiltInAgent reaches its provider through the AI SDK, so there is no response body to wrap.
• A Bot that never sends response headers. fetch has not resolved, so no body exists yet; that needs a connect-level timeout.
• A stall that lands mid-frame. The injected event follows a half-flushed data: line and the parser rejects both. Forbidden to read bytes, the guard cannot see this, and the same frame fails identically with the watch off.
• The protobuf framing. Closed cleanly rather than written into, since SSE bytes would corrupt a merely slow run; it ends in the client's own wording.
• Extra rows per wedged turn. The runtime opens runs of its own against the same Bot for thread naming, each a separate stream.
• Reporting lag. The sweep runs at a quarter of the timeout, capped at a second: 5604ms recorded against a 5000ms setting.
• No retry or resume. The person is told and asked to ask again.
• No rendering test for either line. The helpers that pick the wording are tested; the drawing was verified by reading and against a live server.

Merge notes

Single-hunk additions to server/src/audit.ts, the admin audit page, server/src/config.ts, server/src/index.ts and .env.example will conflict textually with policy-ask-a-person, repeated-action-detector and routines-and-webhook-triggers, and all resolve by keeping both entries. queue-and-steer-a-busy-bot needs more care: it edits the same three channel components and adds its own props and rendering in the same hunk as the shimmer. Both sets of props must survive, and so must the rule added here, that the stopped line replaces the shimmer rather than sitting beside it. Separately, mountCopilotRuntime gained a required stallGuard before its defaulted basePath.

Verification

format and format:check over 295 files, no fixes applied; lint exit 0 with 24 warnings, all pre-existing and elsewhere; typecheck exit 0 for app, server and worker; build exit 0. bun run test gives 635 pass, 5 skip, 0 fail across 73 files, 41 more than main's 594; the skips are the pre-existing smoke tests gated on OPENBOT_SMOKE.

server/tests/turn-watchdog.test.ts pins the activity rule on an injected clock, including that a paused handover is not charged to the Bot. server/tests/stall-guard.test.ts runs real streams through the wrapper: the sentence and row a silent Bot produces, bytes unchanged when the reader waits five times the timeout before its first read, and recovery completing while the audit insert never settles. app/tests/stopped-turn.test.ts and app/tests/audit-silence.test.ts cover the wording of the two lines a person reads. Both defect fixes carry a test confirmed failing beforehand.

The whole path was also driven against a live server with a silent stub Bot, which produced the RUN_ERROR in the thread's events, a cancelled upstream stream, and the audit row.

[davidmckayv]

Reviewed against the code and run locally. Full suite passes on a clean database (bun run test:ci), format, lint and typecheck clean.

No security concerns: this touches no auth boundary and nothing it adds can be reached without going through the gateway.

[davidmckayv]

Re-approving after the rebase. 686 pass / 0 fail locally.

The conflict with #16 that I stopped short of resolving is gone, and the merge kept both sides: the stopped-turn notice and the queued-message rendering coexist rather than one displacing the other.

Adds no migration, so it is unaffected by the 0001 numbering across #15, #18 and #20.