perf(runner): collapse heartbeat + events hot paths to single RPCs#6
Open
pat-lewczuk wants to merge 2 commits into
Open
perf(runner): collapse heartbeat + events hot paths to single RPCs#6pat-lewczuk wants to merge 2 commits into
pat-lewczuk wants to merge 2 commits into
Conversation
The /api/runner/heartbeat and /api/runner/runs/:id/events endpoints were
firing 4 and ~30-60 sequential Postgres statements per call respectively.
Under multi-runner load this saturated the PgBouncer pool, producing the
bimodal 200ms / 10-16s tail latency we hit on dev (and would hit harder
in prod). Both paths now do a single round-trip:
* runner_heartbeat(p_runner_id, p_status) — one CTE pipeline: bump
last_heartbeat_at/status + return cancel_job_ids/pause_run_ids.
Replaces 4 sequential statements.
* ingest_runner_events(p_run_id, p_workspace_id, p_events jsonb) —
one set-based insert opens agent_runs rows from step-start, one
upsert closes them from step-end (needs the new
agent_runs_run_step_iter_uniq constraint), one insert appends every
event to agent_run_events, one atomic UPDATE rolls up tokens_used
+ current_step_id on workflow_runs. The tokens_used roll-up is now
`tokens_used = tokens_used + delta`, fixing a read-modify-write
race on concurrent batches.
Also drops the duplicate touch_runner_heartbeat from /api/runner/jobs —
the daemon's dedicated heartbeat is the single source of truth now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
Lands the in-flight "flows" feature — a UI for declarative skill chains
(name + ordered { skill, argsTemplate, stopChainIfContains } steps) that
run through the existing workflow engine and surface in the cockpit.
Includes:
- migration 0021_flows: flows table + auto-triggers
- core/workflows/flow-runner: turns a FlowRow into a Workflow<FlowBlackboard>;
threads {{input}} / {{previousTaskId}} / {{previousPullRequest*}} into each
step's user prompt; parses PR_URL= / PR_NUMBER= markers between steps
- core/workflows: new unparsedCommentSection callback fires on the
onNoParse success path so flow steps actually populate the living
comment (fixes "(no sections yet)" / "Done." bug observed on the
first end-to-end run on issue #2034)
- gui/app/workflows: /workflows page + flow card editor + starter
templates dropdown; the autofix template is decomposed into 5 focused
steps (verify-in-repo → root-cause → fix → open-pr → review) so each
step gets its own token budget instead of one mega-step blowing past
the 2.5M cap
- gui/app/issues: "Run flow for issue" modal entry point
- webhook + dispatch + runner: route incoming flow runs through the
engine
- docs: open-mercato ephemeral integration guide for downstream verify
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the bimodal 200ms / 10–16s tail latency observed on
/api/runner/heartbeatand/api/runner/runs/:id/eventsunder multi-runner load — these endpoints were firing 4 and ~30–60 sequential Postgres statements per request, saturating the PgBouncer pool. Both paths now do a single RPC round-trip.runner_heartbeat(p_runner_id, p_status)RPC. One CTE pipeline updatesrunners.last_heartbeat_at/statusand returnscancel_job_ids+pause_run_idsin one call (was: 4 sequential statements).ingest_runner_events(p_run_id, p_workspace_id, p_events jsonb)RPC. One set-based insert opensagent_runsrows fromstep-start, one upsert closes them fromstep-end, one insert appends every event toagent_run_events(withORDER BY ordinalityso identity ordering stays stable for the cockpit), one atomic update rolls uptokens_used+current_step_idonworkflow_runs(was: ~30–60 sequential statements per 25-event batch).tokens_used = tokens_used + deltainstead of select-then-update — fixes a pre-existing read-modify-write race on concurrent batches.agent_runs_run_step_iter_uniqon(workflow_run_id, step_id, iteration)so the events RPC canON CONFLICT … DO UPDATE. The combination was logically unique already./api/runner/jobs: the daemon's dedicated heartbeat is the single source of truth — thetouch_runner_heartbeatcall per claim tick was pure overhead.Expected impact
Backward compatibility
RunnerEvent/ heartbeat reply shapes are identical.touch_runner_heartbeatSQL function is still in place (used by nothing now, but kept so this PR is migration-only-additive). Can be dropped in a follow-up.0020_runner_perf.sqlis additive — no destructive DDL. The unique constraint will fail if existing rows have duplicates; a staging DB should be checked before applying (in this codebase the inserts have always been logically unique, so this shouldn't fire).What this does NOT change (deliberately)
This is items #1 + #2 from a larger latency evaluation. Not in this PR (see the eval for the full list):
agent_run_events.Test plan
yarn workspace @cezar/gui run typecheck— passes locally.0020_runner_perf.sqlto a staging Supabase and re-run the heartbeat + events RPCs by hand (select * from runner_heartbeat('<runner-id>', 'online')and a syntheticingest_runner_events(...)call).@cezar/runnerdaemon against staging and confirm heartbeats/events still produce the sameagent_runs/agent_run_events/workflow_runsrows as before.ORDER BY ordinalityclause is what keeps the identity sequence monotonic).next devlogs for the heartbeat/events POST durations — should be back in the ~200ms band.🤖 Generated with Claude Code