Skip to content

Implement eval follow-ups from the #1140 run - #26

Merged
jirispilka merged 1 commit into
mainfrom
implement-eval-followups-20
Aug 6, 2026
Merged

Implement eval follow-ups from the #1140 run#26
jirispilka merged 1 commit into
mainfrom
implement-eval-followups-20

Conversation

@jirispilka

Copy link
Copy Markdown
Collaborator

Closes #20. Five guardrails, each pinned by a contract test citing the run's measured cost:

  • Decay rule: a round whose open findings are all doc/comment-only is proposed as pr-note instead of another fix round (iters 5-9 burned ~1.2M tokens polishing comments and introducing new wrong ones). Replaces the step-7 light re-review path, now deleted.
  • Reviewer Questions: entries route to the implementer only to verify (recorded in iter-N/question-verification.md), never as fixes to apply (~500k tokens on a built-then-reverted suggestion).
  • followups reads the repo's open issues and PRs so dispositions cite existing issues instead of guessing or duplicating.
  • Dispatch start times land in _progress.md; status unknown reports state time since dispatch (a dead implementer surfaced ~9h late).
  • New files enter diff.patch via git diff --no-index /dev/null ; git diff is silent for untracked paths.

Closes #20. Five guardrails, each pinned by a contract test citing the
run's measured cost:

- Decay rule: a round whose open findings are all doc/comment-only is
  proposed as pr-note instead of another fix round (iters 5-9 burned
  ~1.2M tokens polishing comments and introducing new wrong ones).
  Replaces the step-7 light re-review path, now deleted.
- Reviewer Questions: entries route to the implementer only to verify
  (recorded in iter-N/question-verification.md), never as fixes to
  apply (~500k tokens on a built-then-reverted suggestion).
- followups reads the repo's open issues and PRs so dispositions cite
  existing issues instead of guessing or duplicating.
- Dispatch start times land in _progress.md; status unknown reports
  state time since dispatch (a dead implementer surfaced ~9h late).
- New files enter diff.patch via git diff --no-index /dev/null <file>;
  git diff <ref> is silent for untracked paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jirispilka jirispilka self-assigned this Jul 31, 2026
@jirispilka
jirispilka merged commit 1fb2003 into main Aug 6, 2026
2 checks passed
@jirispilka
jirispilka deleted the implement-eval-followups-20 branch August 6, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore: Follow-ups from evaluating the apify-mcp-server #1140 run

2 participants