feat(compose-agent): write-mode acceptance evals + CI validation (4.4.0)

[hamen]

What

Adds compose-agent's own write-mode acceptance evals — the last follow-up flagged across the 4.3.x cross-reviews. Until now the authoring skill had no evals; its new rules were only exercised through jetpack-compose-audit's cases, which grade an audit of existing code, not code the skill writes.

Changes

File	Change
skills/compose-agent/evals/evals.json	New. 5 write-mode cases. 0–2 mirror the 4.3.x audit rules (cross-phase back-write, Strong Skipping false leads, snapshot self-invalidation); 3–4 cover phase-correct reads + lifecycle-aware Flow collection
bin/ci	Validates every skills/<name>/evals/evals.json (JSON, unique numeric ids, required fields, non-empty expectations) + asserts the compose-agent suite keeps covering the three authoring rules
skills/compose-agent/SKILL.md	New Acceptance Evals section
versions	compose-agent → 4.4.0; jetpack-compose-audit unchanged at 4.3.2
CHANGELOG.md, README.md, docs/release-notes-4.4.0.md	4.4.0 notes

Why the CI hook

The point of these evals is to keep the authoring path in lockstep with the audit path. The CI assertion (cross-phase / Strong-Skipping-false-lead / self-invalidation coverage) makes that guarantee mechanical — a future edit can't quietly drop a rule from the suite. Negative-tested: bin/ci rejects an eval with empty expectations.

Note

Evals are acceptance data, not a runner (same model as the audit suite) — run a model with the skill loaded against each prompt and check every expectation.

Validation

bin/ci ✅ (incl. the new validator, README link check, version match).

🤖 Generated with Claude Code

https://claude.ai/code/session_016ysskXsKpwEbs1n9CX1ScY

[hamen]

🟠 Antigravity review (automated cross-review)

This PR does a great job adding write-mode acceptance evals and structurally validating them in CI. The Compose guidance is spot-on.

However, the CI checks meant to prevent rule drift are overly permissive and easily bypassed.

Should-fix

• CI rule enforcement is trivially bypassed by the suite description. In bin/ci, needle.test(composeEvalsText) runs against the entire raw file string. Because the words cross-phase, Strong Skipping, and self-invalidation are all listed in the top-level "description" field of evals.json, the first and third regexes will pass even if the actual test cases are entirely deleted.
• Global regex matching for "Strong Skipping false lead". The regex /Strong Skipping[\s\S]*remember|.../ spans the entire file globally. It will pass as long as "Strong Skipping" appears in Case 0 and "remember" appears in Case 2, meaning Case 1 could be dropped without failing CI.
  • Fix for both: Instead of regexing the raw file text, use the parsed JSON to assert that at least one individual case matches the rule.

    const composeSuite = JSON.parse(readFileSync(composeEvalsPath, 'utf8'));
    // Inside the loop:
    if (!composeSuite.evals.some(e => needle.test(JSON.stringify(e)))) { ... }

Nit

• Redundant file read: bin/ci re-reads compose-agent/evals/evals.json via readFileSync at the end, even though it was already read and parsed in the for (const evalFile of ...) loop just above it.
• Shallow validation: The CI script verifies that item.expectations is a non-empty array but doesn't verify that its contents are actually non-empty strings.

_{via pr-review-relay · author: claude}

[hamen]

🟢 Codex review (automated cross-review)

Blocker: None.

Should-fix:

• bin/ci:215: the Strong Skipping coverage gate only requires Strong Skipping plus memoiz, so it locks in the auto-memoized-lambda false lead but not the remember(i) / remember(index) pure-expression false lead that this PR says case 1 mirrors and currently expects in evals.json:24. A future edit could remove that expectation and CI would still pass. Add an explicit per-case token check or coverage tags for both false-lead subcases.

Nit: None.

Validation: read gh pr view 13, gh pr diff 13, touched files, and the audit eval context. bash bin/ci passes.

_{via pr-review-relay · author: claude}

[hamen]

🔵 Cursor review (automated cross-review)

PR #13 Review — feat(compose-agent): write-mode acceptance evals + CI validation (4.4.0)

Solid, focused release. The write-mode evals (0–2) mirror audit cases 12–14 in the right direction, cases 3–4 align with existing skill docs, versioning/docs are consistent, and CI already passes. No blockers.

---

Blocker

None.

---

Should-fix

1. Audit-side lockstep is weaker than compose-side. CI regex-checks compose-agent cases for the three 4.3.x rules (plus distinct-case + remember(index) subcase), but for jetpack-compose-audit it only asserts that ids 12, 13, 14 exist. Someone could hollow out those audit cases and CI would still pass. If “bidirectional lockstep” is the goal, mirror at least keyword/subcase checks on the audit suite too (or document that compose-side is the enforced side).

2. falseLeadCase lookup is loosely coupled. After the distinct-case loop, the extra check does composeSuite.evals.find(item => /memoiz/i.test(...)) instead of reusing the case already assigned to the Strong Skipping rule. Today only case 1 matches, but a future case mentioning “memoization” could make this fail or check the wrong case. Tie it to the matched id from the Strong Skipping iteration.

---

Nit

1. Case 3 expectation is hard to grade mechanically — “ProgressBar does not recompose on every frame” needs Layout Inspector / compiler reports; fine for manual acceptance evals, worth noting in release docs if you ever automate grading.

2. Eval schema is CI-only — structural rules (files array required but may be empty, etc.) live in bin/ci but aren’t documented in docs/ or SKILL.md. Minor drift risk for contributors adding a third skill suite.

3. Case 1 doesn’t require the SSM-off / @DontMemoize caveat that audit case 13 includes. Acceptable given the prompt says SSM-on; optional extra expectation if you want parity with audit case 13’s last bullet.

---

Verdict: Good to merge — closes the 4.3.x follow-up cleanly; the should-fix items are hardening, not correctness bugs in what ships today.

_{via pr-review-relay · author: claude}