fix: run heartbeat LLM check when instructions aren't GitHub-only - #433
Open
dimavedenyapin wants to merge 1 commit into
Open
fix: run heartbeat LLM check when instructions aren't GitHub-only#433dimavedenyapin wants to merge 1 commit into
dimavedenyapin wants to merge 1 commit into
Conversation
The pre-flight fast path treated any GitHub PR/issue URL in heartbeat.md as covering the whole file. When the linked PR was quiet, runPreflightChecks returned 'no_changes' and the LLM was skipped entirely — so free-text instructions in the same file (e.g. "Monitor if any new transactions appeared in the profiler") were never actually checked. In production this logged "Pre-flight: no changes detected (LLM skipped)" 12 times in a row while the non-GitHub check silently never ran. Pre-flight may now only short-circuit when every check line in heartbeat.md carries a GitHub URL it can verify with `gh api`. Bodies of purely informational sections (## Current Status, ## Latest Finding, …) are skipped, so GitHub-only heartbeats keep the cheap fast path. The check runs before any `gh` call, so mixed files no longer pay for the API round-trips either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A heartbeat whose instructions are free text never got checked by the LLM — because one unrelated PR link elsewhere in the same file silenced the whole file.
runPreflightChecks()treats the presence of any GitHub URL inheartbeat.mdas authority over the entire file. Ifgh apireports no new comments / reviews / CI failures / conflicts on that PR, it returnsno_changes, andrunHeartbeat()takes that as "nothing to do" and skips the LLM session entirely. Every non-GitHub instruction in the file is then silently never evaluated.The reported task's
heartbeat.mdis 5.4 KB, and contains exactly one GitHub URL — buried in a findings note, not even a check:None of the actual checks are GitHub checks. All of them were being skipped. Running the real file through the current code confirms it:
heartbeat_logsfor that task shows the fast path swallowing 8 consecutive runs over ~8 hours:(That log line is emitted only on
no_changes, which is unreachable when zero URLs are extracted — so this is proof the file had the link, not an inference.)The single escape hatch,
requiresLlmCurrentStateChecks(), only matches/requested changes/, so it does not help.Note: a heartbeat containing no GitHub URL at all already works correctly — pre-flight returns
inconclusiveand the LLM runs. The bug needs a URL to be present somewhere in the file.Fix
Pre-flight may short-circuit only when every check line carries a GitHub URL it can verify with
gh api(preflightCoversAllChecks). Anything else returnsinconclusive, which routes to the LLM exactly as before.To keep the cost optimisation for genuinely GitHub-only heartbeats, bodies of purely informational sections (
## Latest Findings,## Current Status,## Notes, …) are not treated as checks — agents write those to record context, not instructions. That is also why the incidental PR link above no longer counts as coverage. Headings, horizontal rules, blockquotes and fenced code are skipped too.The guard runs before any
ghcall, so mixed files no longer pay for API round-trips whose result would be discarded.Test plan
no_changesbefore,inconclusivenow; verified failing against the unfixed code withexpected 'no_changes' to be 'inconclusive'heartbeat.md(see output above)no_changes, and a failed check-run still returnschanges_detectedghis stubbed in tests, so pre-flight no longer hits the networkpnpm test:main— 978 passed / 51 filespnpm typecheckclean;eslintclean on both files🤖 Generated with Claude Code