Skip to content

fix: run heartbeat LLM check when instructions aren't GitHub-only - #433

Open
dimavedenyapin wants to merge 1 commit into
mainfrom
fix/heartbeat-preflight-skips-non-github-checks
Open

fix: run heartbeat LLM check when instructions aren't GitHub-only#433
dimavedenyapin wants to merge 1 commit into
mainfrom
fix/heartbeat-preflight-skips-non-github-checks

Conversation

@dimavedenyapin

@dimavedenyapin dimavedenyapin commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

A heartbeat whose instructions are free text never got checked by the LLM — because one unrelated PR link elsewhere in the same file silenced the whole file.

runPreflightChecks() treats the presence of any GitHub URL in heartbeat.md as authority over the entire file. If gh api reports no new comments / reviews / CI failures / conflicts on that PR, it returns no_changes, and runHeartbeat() takes that as "nothing to do" and skips the LLM session entirely. Every non-GitHub instruction in the file is then silently never evaluated.

The reported task's heartbeat.md is 5.4 KB, and contains exactly one GitHub URL — buried in a findings note, not even a check:

## Latest Findings
- The PO matching aggregation wrapped `_id`, `issuerId`, ... Fix: https://github.com/peakflo/upload-functions/pull/9685.

## Heartbeat Checks
- [ ] Create `issuerId_1_scheduledDate_1` on `billing.bills`, then verify ...
- [ ] Recheck the next hourly Query Stats window ...
- [ ] Rewrite and test the AIExtractor company-alias aggregation ...
- [ ] Replace the repeated tax-embedding anti-join ...

None of the actual checks are GitHub checks. All of them were being skipped. Running the real file through the current code confirms it:

GITHUB_URLS_FOUND        = [{"owner":"peakflo","repo":"upload-functions","type":"pull","number":9685}]
OLD_CODE_TAKES_FAST_PATH = true
NEW_CODE_TAKES_FAST_PATH = false

heartbeat_logs for that task shows the fast path swallowing 8 consecutive runs over ~8 hours:

2026-07-31T02:04:14Z | ok | Pre-flight: no changes detected (LLM skipped)
2026-07-31T00:03:39Z | ok | Pre-flight: no changes detected (LLM skipped)
2026-07-30T22:02:44Z | ok | Pre-flight: no changes detected (LLM skipped)
...

(That log line is emitted only on no_changes, which is unreachable when zero URLs are extracted — so this is proof the file had the link, not an inference.)

The single escape hatch, requiresLlmCurrentStateChecks(), only matches /requested changes/, so it does not help.

Note: a heartbeat containing no GitHub URL at all already works correctly — pre-flight returns inconclusive and the LLM runs. The bug needs a URL to be present somewhere in the file.

Fix

Pre-flight may short-circuit only when every check line carries a GitHub URL it can verify with gh api (preflightCoversAllChecks). Anything else returns inconclusive, which routes to the LLM exactly as before.

To keep the cost optimisation for genuinely GitHub-only heartbeats, bodies of purely informational sections (## Latest Findings, ## Current Status, ## Notes, …) are not treated as checks — agents write those to record context, not instructions. That is also why the incidental PR link above no longer counts as coverage. Headings, horizontal rules, blockquotes and fenced code are skipped too.

The guard runs before any gh call, so mixed files no longer pay for API round-trips whose result would be discarded.

Test plan

  • Regression test reproduces the production shape (quiet PR link + free-text check) — returned no_changes before, inconclusive now; verified failing against the unfixed code with expected 'no_changes' to be 'inconclusive'
  • Verified against the actual 5.4 KB production heartbeat.md (see output above)
  • Control tests confirm the fast path survives — a GitHub-only heartbeat with a quiet PR still returns no_changes, and a failed check-run still returns changes_detected
  • gh is stubbed in tests, so pre-flight no longer hits the network
  • pnpm test:main — 978 passed / 51 files
  • pnpm typecheck clean; eslint clean on both files

🤖 Generated with Claude Code

The pre-flight fast path treated any GitHub PR/issue URL in heartbeat.md as
covering the whole file. When the linked PR was quiet, runPreflightChecks
returned 'no_changes' and the LLM was skipped entirely — so free-text
instructions in the same file (e.g. "Monitor if any new transactions appeared
in the profiler") were never actually checked. In production this logged
"Pre-flight: no changes detected (LLM skipped)" 12 times in a row while the
non-GitHub check silently never ran.

Pre-flight may now only short-circuit when every check line in heartbeat.md
carries a GitHub URL it can verify with `gh api`. Bodies of purely
informational sections (## Current Status, ## Latest Finding, …) are skipped,
so GitHub-only heartbeats keep the cheap fast path. The check runs before any
`gh` call, so mixed files no longer pay for the API round-trips either.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant