Skip to content

perf(runner): collapse heartbeat + events hot paths to single RPCs#6

Open
pat-lewczuk wants to merge 2 commits into
mainfrom
perf/runner-rpc-hotpaths
Open

perf(runner): collapse heartbeat + events hot paths to single RPCs#6
pat-lewczuk wants to merge 2 commits into
mainfrom
perf/runner-rpc-hotpaths

Conversation

@pat-lewczuk

Copy link
Copy Markdown
Member

Summary

Fixes the bimodal 200ms / 10–16s tail latency observed on /api/runner/heartbeat and /api/runner/runs/:id/events under multi-runner load — these endpoints were firing 4 and ~30–60 sequential Postgres statements per request, saturating the PgBouncer pool. Both paths now do a single RPC round-trip.

  • Heartbeat: new runner_heartbeat(p_runner_id, p_status) RPC. One CTE pipeline updates runners.last_heartbeat_at/status and returns cancel_job_ids + pause_run_ids in one call (was: 4 sequential statements).
  • Events ingest: new ingest_runner_events(p_run_id, p_workspace_id, p_events jsonb) RPC. One set-based insert opens agent_runs rows from step-start, one upsert closes them from step-end, one insert appends every event to agent_run_events (with ORDER BY ordinality so identity ordering stays stable for the cockpit), one atomic update rolls up tokens_used + current_step_id on workflow_runs (was: ~30–60 sequential statements per 25-event batch).
  • Atomic token roll-up: tokens_used = tokens_used + delta instead of select-then-update — fixes a pre-existing read-modify-write race on concurrent batches.
  • New unique constraint agent_runs_run_step_iter_uniq on (workflow_run_id, step_id, iteration) so the events RPC can ON CONFLICT … DO UPDATE. The combination was logically unique already.
  • Drops duplicate heartbeat from /api/runner/jobs: the daemon's dedicated heartbeat is the single source of truth — the touch_runner_heartbeat call per claim tick was pure overhead.

Expected impact

  • Heartbeat p95 should drop from ~10–16s under contention back into the ~50–200ms band.
  • Events POST p95 should drop ~10–20× since pool occupancy time per request goes from ~30 statements to 1.
  • The architecture scales to dozens of runners — the previous bottleneck wasn't throughput, it was per-request statement count × pool size.

Backward compatibility

  • Wire format unchanged — the runner's RunnerEvent / heartbeat reply shapes are identical.
  • The legacy touch_runner_heartbeat SQL function is still in place (used by nothing now, but kept so this PR is migration-only-additive). Can be dropped in a follow-up.
  • Migration 0020_runner_perf.sql is additive — no destructive DDL. The unique constraint will fail if existing rows have duplicates; a staging DB should be checked before applying (in this codebase the inserts have always been logically unique, so this shouldn't fire).

What this does NOT change (deliberately)

This is items #1 + #2 from a larger latency evaluation. Not in this PR (see the eval for the full list):

  • Push-based pause/cancel via SSE instead of heartbeat polling.
  • Per-job concurrency in the runner daemon.
  • Range-partition / TTL on agent_run_events.
  • Realtime backpressure.

Test plan

  • yarn workspace @cezar/gui run typecheck — passes locally.
  • Apply 0020_runner_perf.sql to a staging Supabase and re-run the heartbeat + events RPCs by hand (select * from runner_heartbeat('<runner-id>', 'online') and a synthetic ingest_runner_events(...) call).
  • Spin up the @cezar/runner daemon against staging and confirm heartbeats/events still produce the same agent_runs / agent_run_events / workflow_runs rows as before.
  • Trigger a multi-step autofix and confirm the cockpit live view still updates in order (the ORDER BY ordinality clause is what keeps the identity sequence monotonic).
  • Watch next dev logs for the heartbeat/events POST durations — should be back in the ~200ms band.

🤖 Generated with Claude Code

The /api/runner/heartbeat and /api/runner/runs/:id/events endpoints were
firing 4 and ~30-60 sequential Postgres statements per call respectively.
Under multi-runner load this saturated the PgBouncer pool, producing the
bimodal 200ms / 10-16s tail latency we hit on dev (and would hit harder
in prod). Both paths now do a single round-trip:

  * runner_heartbeat(p_runner_id, p_status) — one CTE pipeline: bump
    last_heartbeat_at/status + return cancel_job_ids/pause_run_ids.
    Replaces 4 sequential statements.
  * ingest_runner_events(p_run_id, p_workspace_id, p_events jsonb) —
    one set-based insert opens agent_runs rows from step-start, one
    upsert closes them from step-end (needs the new
    agent_runs_run_step_iter_uniq constraint), one insert appends every
    event to agent_run_events, one atomic UPDATE rolls up tokens_used
    + current_step_id on workflow_runs. The tokens_used roll-up is now
    `tokens_used = tokens_used + delta`, fixing a read-modify-write
    race on concurrent batches.

Also drops the duplicate touch_runner_heartbeat from /api/runner/jobs —
the daemon's dedicated heartbeat is the single source of truth now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@supabase

supabase Bot commented May 24, 2026

Copy link
Copy Markdown

This pull request has been ignored for the connected project mycifnmzgmaklyhcyyyw because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.


Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

Lands the in-flight "flows" feature — a UI for declarative skill chains
(name + ordered { skill, argsTemplate, stopChainIfContains } steps) that
run through the existing workflow engine and surface in the cockpit.

Includes:
- migration 0021_flows: flows table + auto-triggers
- core/workflows/flow-runner: turns a FlowRow into a Workflow<FlowBlackboard>;
  threads {{input}} / {{previousTaskId}} / {{previousPullRequest*}} into each
  step's user prompt; parses PR_URL= / PR_NUMBER= markers between steps
- core/workflows: new unparsedCommentSection callback fires on the
  onNoParse success path so flow steps actually populate the living
  comment (fixes "(no sections yet)" / "Done." bug observed on the
  first end-to-end run on issue #2034)
- gui/app/workflows: /workflows page + flow card editor + starter
  templates dropdown; the autofix template is decomposed into 5 focused
  steps (verify-in-repo → root-cause → fix → open-pr → review) so each
  step gets its own token budget instead of one mega-step blowing past
  the 2.5M cap
- gui/app/issues: "Run flow for issue" modal entry point
- webhook + dispatch + runner: route incoming flow runs through the
  engine
- docs: open-mercato ephemeral integration guide for downstream verify

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant