fix(overseer): tolerate trailing text after a health-review decision's JSON - #4938
Open
rysweet wants to merge 1 commit into
Open
Conversation
…s JSON
The Overseer's agentic HEALTH-REVIEW rail parses plain-text DECISION markers
(`LAUNCH_RECIPE={json}` / `ESCALATE_GOAL={json}`) into the typed interventions it
drives through the SAME gated capability path every other fix uses. A prior
hardening made the parser tolerant of LEADING markdown decoration, but the JSON
was still handed straight to `serde_json::from_str`, which REJECTS any trailing
non-whitespace after the object.
So a well-formed decision an agent wrote as
`LAUNCH_RECIPE={...} — fixes the crash-loop` parsed to ZERO interventions while
the terminal `HEALTH_REVIEW_COMPLETE=` marker still parsed "successfully" — the
degraded-pass escalation ladder never fired, and a real crash-loop's remediation
was lost with NO signal. That is the exact silent-drop the leading-decoration fix
closed, but on the trailing side, and it directly defeats "DRIVE remediation
through EXISTING capabilities".
Fix (thin rail, mechanical, fail-closed): add `extract_leading_json_object`, a
balanced-brace scanner that respects JSON strings + escapes, used by
`parse_launch_decision` / `parse_escalate_decision` before serde. It:
- extracts the leading `{…}` object and discards any trailing clause (the
trailing text never leaks into the parsed brief / operator-facing fields);
- returns `None` for a payload that does not start with `{` or whose braces never
balance, so the caller falls back to the raw payload — which serde then rejects
— rather than half-parsing a truncated/broken decision (fail-closed);
- never edits the JSON it returns; serde still validates every required field;
- returns the FIRST object on a concatenation, so a trailing `{...}` cannot
smuggle a second decision.
No behavior change for exact-JSON payloads (byte-identical parse). No prompt,
capability, gate, or output-contract change; NO new imperative failure-plumbing
(no `record_step_failure` at an origin site, no consecutive-failure counter, no
N-identical-failure threshold) — reasoning stays broad, authorization narrow.
Tests: 6 new unit tests (trailing text after launch/escalate, leading decoration
+ trailing text composed, `}` inside a string value, unbalanced/truncated skipped
fail-closed, extractor basics). Pinned as required invariants in the outside-in
gadugi scenario `tests/gadugi/overseer-health-review.sh` and covered by a new QA
scenario `tests/qa-scenarios/overseer-health-review-trailing-text.yaml`. Concept
doc updated with the trailing-side symmetry.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Overseer's agentic HEALTH-REVIEW rail ([standing] goal) reads the
journal, reasons to a small set of typed DECISION markers
(
LAUNCH_RECIPE=/ESCALATE_GOAL=/HEALTH_REVIEW_COMPLETE=), and the thin Rustrail (
src/overseer/health_review.rs) routes each onto the same gatedcapability path every other fix uses. A prior hardening made the parser
tolerant of leading markdown decoration, but each decision's JSON payload was
still handed straight to
serde_json::from_str, which rejects any trailingnon-whitespace after the object.
So a well-formed decision an agent wrote as
LAUNCH_RECIPE={...} — fixes the crash-loopparsed to zero interventionswhile the terminal
HEALTH_REVIEW_COMPLETE=marker still parsed "successfully" —the degraded-pass escalation ladder never fired and a real crash-loop's
remediation was lost with no signal. That is the exact silent-drop the
leading-decoration fix closed, but on the trailing side, and it directly defeats
"DRIVE remediation through EXISTING capabilities".
Fix (thin rail, mechanical, fail-closed)
Add
extract_leading_json_object, a balanced-brace scanner that respects JSONstrings + escapes, used by
parse_launch_decision/parse_escalate_decisionbefore serde. It:
{…}object and discards any trailing clause (trailingtext never leaks into the parsed brief / operator-facing fields);
Nonefor a payload that does not start with{or whose braces neverbalance, so the caller falls back to the raw payload — which serde then rejects
— rather than half-parsing a truncated/broken decision (fail-closed);
{...}cannotsmuggle a second decision.
Byte-identical parse for exact-JSON payloads. No prompt / capability /
gate / output-contract change. Critically, no new imperative failure-plumbing
— no
record_step_failureat an origin site, no consecutive-failure counter, noN-identical-failure threshold. Reasoning stays broad; authorization stays narrow;
the anti-pattern the standing goal forbids is not touched.
Changed surfaces (all internal)
src/overseer/health_review.rsextract_leading_json_object+ wired into both parse fns + doc bullet + 6 unit teststests/gadugi/overseer-health-review.shtests/qa-scenarios/overseer-health-review-trailing-text.yamldocs/concepts/overseer-agentic-health-review.mdNo user-facing surface changed (the marker contract, prompt, and operator
notification text are unchanged); this is a parser-robustness fix on an internal
rail, plus the concept doc that already documents that rail.
Merge-ready evidence
1. QA / gadugi scenarios
tests/qa-scenarios/overseer-health-review-trailing-text.yamldrives the hermetic unit tests + full overseer suite (YAML validated: 5 keys, 6 steps).
tests/gadugi/overseer-health-review.shpins the 6 newtests via
require_testand was run green:2. Docs
docs/concepts/overseer-agentic-health-review.mdupdated with the trailing-sidesymmetry paragraph (+
last_updated). All other changed surfaces areinternal-only (parser rail + test harness).
3. Quality-audit (3 SEEK→VALIDATE→FIX cycles, ended clean)
start with
{, returns atdepth==0), correct UTF-8 slicing (i + len_utf8),string/escape state respected. VALIDATED by the 6 new tests incl.
extract_leading_json_object_basics. No fix needed.(returns the FIRST object; trailing discarded, never re-parsed as a marker);
trailing text never leaks into
task_description/next_step; unbalanced →None→ serde reject → skipped. VALIDATED by brace-in-string + unbalanced +existing 54 tests. No fix needed.
no
print!/println!; noBridgenaming; byte-identical exact-JSON parse;require_testnames match test fns exactly. VALIDATED by fmt + clippy--all-targets --all-features -D warnings+ fullhealth_reviewfilter acrossall targets. Clean final cycle — zero critical/high, zero medium
correctness/security findings.
4. CI-relevant gates (run locally, all green)
cargo fmt --all -- --check→ cleancargo clippy --release --no-deps -- -D warnings(pre-commit) → cleancargo clippy --all-targets --all-features --locked -- -D warnings→ cleancargo test --lib overseer::health_review::tests→ 35 passed, 0 failed(incl. the 6 new tests)
cargo test --lib overseer::→ 749 passed, 0 failedcargo test --all-features --locked health_review(all targets) → ok, 0 failed6. Focused diff
4 files, +215/−7, all within the health-review rail + its test harness + its
concept doc. No unrelated edits.