Skip to content

fix(overseer): tolerate trailing text after a health-review decision's JSON - #4938

Open
rysweet wants to merge 1 commit into
mainfrom
engineer/give-the-overseer-an-agentic-health-review-reci-91b52421-1785209607-3dcaf9
Open

fix(overseer): tolerate trailing text after a health-review decision's JSON#4938
rysweet wants to merge 1 commit into
mainfrom
engineer/give-the-overseer-an-agentic-health-review-reci-91b52421-1785209607-3dcaf9

Conversation

@rysweet

@rysweet rysweet commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Summary

The Overseer's agentic HEALTH-REVIEW rail ([standing] goal) reads the
journal, reasons to a small set of typed DECISION markers
(LAUNCH_RECIPE=/ESCALATE_GOAL=/HEALTH_REVIEW_COMPLETE=), and the thin Rust
rail (src/overseer/health_review.rs) routes each onto the same gated
capability path
every other fix uses. A prior hardening made the parser
tolerant of leading markdown decoration, but each decision's JSON payload was
still handed straight to serde_json::from_str, which rejects any trailing
non-whitespace
after the object.

So a well-formed decision an agent wrote as
LAUNCH_RECIPE={...} — fixes the crash-loop parsed to zero interventions
while the terminal HEALTH_REVIEW_COMPLETE= marker still parsed "successfully" —
the degraded-pass escalation ladder never fired and a real crash-loop's
remediation was lost with no signal
. That is the exact silent-drop the
leading-decoration fix closed, but on the trailing side, and it directly defeats
"DRIVE remediation through EXISTING capabilities".

Fix (thin rail, mechanical, fail-closed)

Add extract_leading_json_object, a balanced-brace scanner that respects JSON
strings + escapes, used by parse_launch_decision/parse_escalate_decision
before serde. It:

  • extracts the leading {…} object and discards any trailing clause (trailing
    text never leaks into the parsed brief / operator-facing fields);
  • returns None for a payload that does not start with { or whose braces never
    balance, so the caller falls back to the raw payload — which serde then rejects
    — rather than half-parsing a truncated/broken decision (fail-closed);
  • never edits the JSON it returns (serde still validates every required field);
  • returns the first object on a concatenation, so a trailing {...} cannot
    smuggle a second decision.

Byte-identical parse for exact-JSON payloads. No prompt / capability /
gate / output-contract change. Critically, no new imperative failure-plumbing
— no record_step_failure at an origin site, no consecutive-failure counter, no
N-identical-failure threshold. Reasoning stays broad; authorization stays narrow;
the anti-pattern the standing goal forbids is not touched.

Changed surfaces (all internal)

File Change
src/overseer/health_review.rs new extract_leading_json_object + wired into both parse fns + doc bullet + 6 unit tests
tests/gadugi/overseer-health-review.sh pin the 6 new tests as required invariants
tests/qa-scenarios/overseer-health-review-trailing-text.yaml new outside-in QA scenario
docs/concepts/overseer-agentic-health-review.md document the trailing-side symmetry

No user-facing surface changed (the marker contract, prompt, and operator
notification text are unchanged); this is a parser-robustness fix on an internal
rail, plus the concept doc that already documents that rail.


Merge-ready evidence

1. QA / gadugi scenarios

  • New QA scenario tests/qa-scenarios/overseer-health-review-trailing-text.yaml
    drives the hermetic unit tests + full overseer suite (YAML validated: 5 keys, 6 steps).
  • Gadugi scenario tests/gadugi/overseer-health-review.sh pins the 6 new
    tests via require_test and was run green:
    == (a) recipe-runner json mode surfaces the health-review markers ==
    OK: the recipe-runner seam forwards the health-review decision markers verbatim.
    == (b) in-tree overseer health-review tests ==
    health-review tests passed: 54
    PASS: overseer-health-review scenario ([standing])
    

2. Docs

docs/concepts/overseer-agentic-health-review.md updated with the trailing-side
symmetry paragraph (+ last_updated). All other changed surfaces are
internal-only (parser rail + test harness).

3. Quality-audit (3 SEEK→VALIDATE→FIX cycles, ended clean)

  • Cycle 1 — correctness. Brace-depth scanner: cannot underflow (payload must
    start with {, returns at depth==0), correct UTF-8 slicing (i + len_utf8),
    string/escape state respected. VALIDATED by the 6 new tests incl.
    extract_leading_json_object_basics. No fix needed.
  • Cycle 2 — security / XPIA / fail-closed. Cannot smuggle a second decision
    (returns the FIRST object; trailing discarded, never re-parsed as a marker);
    trailing text never leaks into task_description/next_step; unbalanced →
    None → serde reject → skipped. VALIDATED by brace-in-string + unbalanced +
    existing 54 tests. No fix needed.
  • Cycle 3 — regression / philosophy. Zero new imperative failure-plumbing;
    no print!/println!; no Bridge naming; byte-identical exact-JSON parse;
    require_test names match test fns exactly. VALIDATED by fmt + clippy
    --all-targets --all-features -D warnings + full health_review filter across
    all targets. Clean final cycle — zero critical/high, zero medium
    correctness/security findings.

4. CI-relevant gates (run locally, all green)

  • cargo fmt --all -- --check → clean
  • cargo clippy --release --no-deps -- -D warnings (pre-commit) → clean
  • cargo clippy --all-targets --all-features --locked -- -D warnings → clean
  • cargo test --lib overseer::health_review::tests35 passed, 0 failed
    (incl. the 6 new tests)
  • cargo test --lib overseer::749 passed, 0 failed
  • cargo test --all-features --locked health_review (all targets) → ok, 0 failed
  • pre-push race-subset test → 482 passed, 0 failed

6. Focused diff

4 files, +215/−7, all within the health-review rail + its test harness + its
concept doc. No unrelated edits.

…s JSON

The Overseer's agentic HEALTH-REVIEW rail parses plain-text DECISION markers
(`LAUNCH_RECIPE={json}` / `ESCALATE_GOAL={json}`) into the typed interventions it
drives through the SAME gated capability path every other fix uses. A prior
hardening made the parser tolerant of LEADING markdown decoration, but the JSON
was still handed straight to `serde_json::from_str`, which REJECTS any trailing
non-whitespace after the object.

So a well-formed decision an agent wrote as
`LAUNCH_RECIPE={...} — fixes the crash-loop` parsed to ZERO interventions while
the terminal `HEALTH_REVIEW_COMPLETE=` marker still parsed "successfully" — the
degraded-pass escalation ladder never fired, and a real crash-loop's remediation
was lost with NO signal. That is the exact silent-drop the leading-decoration fix
closed, but on the trailing side, and it directly defeats "DRIVE remediation
through EXISTING capabilities".

Fix (thin rail, mechanical, fail-closed): add `extract_leading_json_object`, a
balanced-brace scanner that respects JSON strings + escapes, used by
`parse_launch_decision` / `parse_escalate_decision` before serde. It:
- extracts the leading `{…}` object and discards any trailing clause (the
  trailing text never leaks into the parsed brief / operator-facing fields);
- returns `None` for a payload that does not start with `{` or whose braces never
  balance, so the caller falls back to the raw payload — which serde then rejects
  — rather than half-parsing a truncated/broken decision (fail-closed);
- never edits the JSON it returns; serde still validates every required field;
- returns the FIRST object on a concatenation, so a trailing `{...}` cannot
  smuggle a second decision.

No behavior change for exact-JSON payloads (byte-identical parse). No prompt,
capability, gate, or output-contract change; NO new imperative failure-plumbing
(no `record_step_failure` at an origin site, no consecutive-failure counter, no
N-identical-failure threshold) — reasoning stays broad, authorization narrow.

Tests: 6 new unit tests (trailing text after launch/escalate, leading decoration
+ trailing text composed, `}` inside a string value, unbalanced/truncated skipped
fail-closed, extractor basics). Pinned as required invariants in the outside-in
gadugi scenario `tests/gadugi/overseer-health-review.sh` and covered by a new QA
scenario `tests/qa-scenarios/overseer-health-review-trailing-text.yaml`. Concept
doc updated with the trailing-side symmetry.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant