Skip to content

CLAUDE.md

Latest commit

 

History

History
68 lines (45 loc) · 18.5 KB

CLAUDE.md

File metadata and controls

68 lines (45 loc) · 18.5 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

CodeCouncil is an AI peer reviewer that watches an AI coding agent (Claude Code) work in real time, flags genuine issues, verifies findings by running a repro before speaking, writes a claims-vs-verified session receipt when the agent declares work done, and rewrites its own review heuristics based on which suggestions turned out to be right — with every rewrite gated by frozen evals and auto-rolled-back on measured regression. Plain-language overview: docs/PROJECT_GUIDE.md.

Commands

python3 -m unittest discover -s tests                    # full suite
python3 -m unittest tests.test_critic                    # one module
python3 -m unittest tests.test_observer.TestTailing      # one class

python3 -m codecouncil [/path/to/repo]   # everything: hooks + observer + critic + reflector (repo defaults to cwd; interactive console: /keys /model /prober /status /verbose)
python3 -m observer /path/to/repo        # observer daemon (--once, --from-start, --interval N)
python3 -m critic /path/to/repo          # critic daemon (--once, --judge-every-beat)
python3 -m reflector /path/to/repo       # reflector daemon (--once, --force-rewrite)
python3 -m reflector.report /path/to/repo  # acceptance per heuristics version
python3 -m hooks.install /path/to/repo   # install Claude Code hooks (idempotent)
python3 -m training.run                  # scripted headless sessions that generate real data (verified clean under CRITIC_CMD stub + a stubbed claude; spawns real, costly claude -p sessions, so point --dir at a scratch repo)
python3 -m evals.run /path/to/repo       # replay frozen cases against every heuristics version
python3 -m evals.extract /path/to/training-repo  # freeze real observation payloads into evals/cases/
python3 -m evals.ab.run --selftest       # zero-spend check: every A/B safety scorer separates its good/bad refs (CI runs this)
python3 -m evals.ab.run --trials N --tier both --arms all  # the A/B benchmark — spawns real claude sessions ($$)
python3 -m evals.ab.rescore /path/to/cc-ab-<ts>  # re-score a persisted A/B run with the current scorer, no new sessions

pipx run --spec 'ruff==0.15.22' ruff check .     # lint — the exact pin CI uses

cd ui && npm run dev                     # dashboard at http://localhost:4700 (COUNCIL_REPO=/path to watch another repo)

Python is stdlib-only by design (3.10+): do not add pip dependencies to observer/critic/reflector/hooks/training/evals — dev tooling (ruff) is CI-only, and its rule set (ruff.toml) is deliberately narrow (E4/E7/E9/F, pinned): the blind-except and try-except-continue patterns it would otherwise flag are intentional fail-open code, not defects. The dashboard (ui/) is React + Vite + Tailwind. CI (.github/workflows/ci.yml) runs the Python suite on 3.10/3.12, tsc --noEmit + the Vite build, the A/B safety-scorer selftest, the pinned ruff lint, and an install.sh smoke test (the one-command curl installer: clones to ~/.codecouncil/app, writes a codecouncil launcher to ~/.local/bin) on every push/PR. AGENTS.md is the contributor contract for AI agents — shorter than this file, same invariants.

Architecture

Four loops communicating only through NDJSON/JSON files in the watched repo's .codecouncil/ directory (gitignored). Each loop is an independent daemon; there are no imports across loop boundaries except small shared utilities (core.store for NDJSON + startup waits, core.redact, observer.events, observer.transcript, critic.agent). codecouncil/ is only a launcher: it preflights the pi/model setup (clear warnings instead of silent per-beat failures), installs hooks, and runs the three loops as subprocesses with prefixed output — filtered for signal (codecouncil/signal_filter.py, pure functions: findings/grades/rewrites are highlighted, idle-beat chatter dropped unless /verbose) and driven by an interactive console (codecouncil/console.py: /keys, /model, /prober, /status work in place, restarting the critic as needed; parse logic is pure and testable, side effects live in Console, and non-TTY stdin never starts a console). CodeCouncil watches its own repo — .codecouncil/ here contains live data, and the hooks are installed on this repo, so the Critic may review your work as you code.

Redaction invariant: every text-bearing field is passed through core.redact.redact() at observer capture time — diffs, untracked/touched file contents, commit subjects + diffs (observer/gitwatch.py), reasoning text and tool-call commands (observer/transcript.py). Nothing downstream (prompts, receipts, harvested eval cases, dashboard) ever holds a raw credential; the «REDACTED:kind» marker itself is taught to the critic as a confirmed secret-in-code finding. If you add a new text field to any event, redact it at capture.

  1. Observer (observer/, event-driven: beats fire when a transcript grows, --interval is only the fallback floor) — pairs intent with reality. Tails Claude Code session transcripts (~/.claude/projects/<munged-path>/*.jsonl, persisted byte offsets in state.json) into reasoning/tool_call events, and snapshots git state into diff events (fingerprinted, emitted only on change; includes capped contents of new untracked files) and commit events (old..new HEAD ranges). Appends to observations.ndjsonl.

  2. Critic (critic/, ~10s beat) — reads new observations; when code actually changed (with a floor between model calls), sends one prompt to a headless pi agent turn asking for PASS-with-reason or ONE suggestion. Before that model call, mechanical screening runs zero-model-cost static checks on the diff's added lines and surfaces signals for the judge to confirm or dismiss: critic/screen.py (injection-pattern strings, unsafe deserialization, weakened tests, unresolvable new imports) and critic/deps.py (typo-suspect/slopsquat-shaped import and dependency-manifest lines, matched against the curated offline import-name snapshot in critic/pkg_names.py) — the documented top AI-code failure modes made visible on purpose. critic/probe.py adds opt-in (--probes) property probes: for a changed function with a docstring promise, one budgeted model turn derives short self-checking scripts and runs them for real, turning a code/docstring contradiction it catches into a finding. Judgment turns can investigate before speaking: JUDGE_TOOLS gives them repo_read/repo_grep/repo_find/repo_ls — path-jailed extension tools (critic/pi_extensions/jail.mjs; pi's builtin read/grep/find/ls are NEVER used for this because they resolve ~/absolute paths outside cwd). Eval replays stay tool-less (hermetic). Model-authored issue/rationale are redacted+capped at parse time (tool output is a live path for repo content into stored artifacts). Every verdict records reviewed_files and saves its judgment packet (case material) — PASS included, so silences are gradeable. Prompts include the diff plus capped current contents of the touched files (touched_contents, budget-reserved before the diff so the diff floor holds). Findings are first verified by running a repro (critic/verify.py) — refuted findings are never delivered. Also runs a task review when the agent declares work done (claims supported? did a test command run — three-state fact backed by the sticky per-session tests_run_at record, so a test run hours earlier still counts) and writes a claims-vs-verified session receipt to receipts/ (critic/receipt.py). Every verdict + its exact prompt goes to suggestions.ndjsonl/prompts/, tagged with heuristics_version and the source session (delivery is session-scoped). Records are ts-stamped at write time (post-verification) so the delivery TTL isn't consumed by model latency. Model calls run on a worker thread; the batch's read offset only commits (committed_offset) after its record lands — a crash replays the batch, a failing batch re-queues up to 3 times then drops loudly. Malformed model replies degrade to PASS but are counted and surfaced (terminal warning + dashboard badge) — a provider format drift must never silently mute the critic.

Council mode (opt-in, critic/main.py): a second, independent model — the prober — gets asked the exact same prompt as the primary. Measured basis (docs/benchmarks/): the primary (NVIDIA Nemotron by default) is a precision anchor, 0 false positives but only 2-of-4 catches on the flaggable bake-off cases; the prober (bake-off measured with OpenRouter openai/gpt-5-mini) has full recall (4-of-4) but 2 false positives on clean changes. merge_council lets the primary's verdict flow through whenever it has one; a prober-only SUGGESTION (primary said PASS) is trusted only after verify.verify_finding reproduces it — with --no-verify there's no repro step, so the prober call is skipped entirely rather than deliver an unverifiable guess (judge_batch's ctx.get("prober") and ctx.get("verify", True) gate). Enable via --prober <model> or COUNCIL_PROBER=<model> env (critic/main.py's resolve_prober: flag > env > off); codecouncil/main.py passes --prober through to the critic subprocess and preflights an openrouter/* prober against OPENROUTER_API_KEY. No council key is ever added to a suggestion row unless a prober is configured — the no-council path is byte-identical to before council mode existed.

  1. Hooks (hooks/) — delivery channel into the coding agent's own context via Claude Code hooks: PostToolUse/UserPromptSubmit inject medium/high suggestions after edits (session-scoped: a finding only reaches the session whose work produced it; untagged rows and task reviews stay repo-wide) and announce new session receipts once; Stop blocks a "done" declaration once for high severity until fixed or rebutted (Stop's hook JSON has no non-blocking context channel — receipts never announce there). Opt-in done-gate (COUNCIL_GATE_SECONDS env or gate_seconds config, off by default): when the critic still has unjudged diff/commit material at Stop, peer_hook.py's _maybe_wait_for_critic holds the "done" declaration open (polled outside any lock, clamped to GATE_SECONDS_MAX=120s) so a finding landing in that window can still be delivered this turn. When the gate genuinely held this Stop, the block-severity floor drops from high-only to medium+ (hooks/logic.py's GATED_BLOCK_SEVERITIES) — a finished one-shot session has no later PostToolUse to inject a medium finding into, so Stop is its only remaining channel. peer_hook.py must fail open (any error → silent exit 0) — preserve that invariant; the delivered.json read→decide→write span is flock-guarded (fail-open if locking fails). Decision rules in hooks/logic.py (pure, no I/O — keep it that way for tests); ledger reserved key "receipts" tracks announced receipts.

  2. Reflector (reflector/, slow beat) — grades each delivered suggestion from post-delivery diffs + reasoning (accepted/rebutted/ignored, model-judged with a deterministic did-the-file-change cross-check; a COUNCIL-REBUTTAL: <reason> line from the coding agent — which the hook text invites — grades rebutted deterministically, no model call) into outcomes.ndjsonl. It also grades the silences: reflector/misses.py detects a fix-shaped commit revising files a recent PASS reviewed (exact-path/basename match + fix-subject, both required — precision first) and grades that PASS missed deterministically; missed packets harvest as must-flag eval cases. missed never enters the acceptance rate (two-signal separation). Rebutted grades additionally distill ONE repo fact each (TASK: DISTILL turn → core/knowledge.py → .codecouncil/knowledge.md, 30-fact cap, directive-filtered as a floor with the persona's facts-not-instructions rule as the real backstop) which the critic reads on every judgment. Suggestions cite the heuristic rule that motivated them ("rule": N against R1.-numbered heuristics); grades carry the rule, reflector.report prints per-rule tables, and rewrite prompts include per-rule graded stats so rewrites are evidence-linked per rule. Then it then rewrites heuristics.md from the grades. Rewrites are a measured control loop: format validation (strict version: N+1 + length) → eval gate (rewrite.gate_candidate scores candidate vs current on the frozen cases — 2×len(cases) real model calls, bounded by backoff: rejected/invalid candidates advance the grade counter so the next attempt waits for fresh grades) → atomic apply with prior versions archived to heuristics-history/. maybe_rollback auto-reverts a version whose in-the-wild acceptance drops below its predecessor's (≥3 grades, revert-once guard, versions only ever increment — a revert is a new version with the old rules). Graded outcomes also harvest new eval cases (reflector/harvest.py → evals/cases-harvested/, deliberately global across watched repos, capped at 40, deduped): accepted findings become must-flag cases, refuted/uncontested-rebutted ones become must-pass cases — so the gate's case set grows from real results. The Critic reads heuristics.md on every call — that file is the thing being self-improved.

Model boundary: all model calls go through critic/agent.py — one non-interactive pi turn (pi -p --no-session --no-tools …, persona via --system-prompt). Judgment turns get no tools; verification turns also get no tools — the model instead writes a self-contained repro SCRIPT, which the harness (not the model) then executes in a throwaway staging directory (critic/verify.py, sharing critic/probe.py's run_script), so repros never touch the watched repo. This replaced an earlier tool-enabled (read,bash) verification turn: the NVIDIA/pi backend frequently emitted its tool calls as literal, never-executed text, which made verification land "inconclusive" and withheld true findings — a script the harness executes itself has no such failure mode. COUNCIL_MODEL=provider/model overrides pi's default; PI_BIN overrides the executable. Set CRITIC_CMD=<executable> to stub the model in tests: it runs as $CRITIC_CMD <prompt-file> <resolved-model>, stdout is the reply. Personas live in critic/persona.md and reflector/persona.md.

critic/agent.py also auto-loads ~/.codecouncil/env (outside any watched repo — a credential placed there can never be committed regardless of which repo CodeCouncil is pointed at) to top up the subprocess environment, and always attaches critic/pi_extensions/nvidia_provider.mjs via pi -e. If COUNCIL_MODEL is unset, the first configured key picks the default model (core.config.KEY_DEFAULT_MODELS, ordered: free NVIDIA-hosted Nemotron first, Anthropic last for decorrelation) — zero pi login required. Model ids in that extension must be NVIDIA's full catalog string (e.g. nvidia/nemotron-3-super-120b-a12b); pi's openai-completions provider sends model.id verbatim as the request's model field, so a shorter id 404s.

Measurement: two independent signals of self-improvement — acceptance rate per heuristics version (reflector/report.py, mirrored exactly by ui/server/council.ts so the dashboard can't diverge from the real metric) and frozen eval cases (evals/cases/*.json + harvested evals/cases-harvested/*.json) replayed against every heuristics version. The signals are deliberately kept separate: the rewrite gate uses eval scores, rollback uses in-the-wild acceptance — never mix them.

A/B benchmark (evals/ab/): does CodeCouncil measurably improve a coding agent's output? Paired design — every (task, trial) runs once per arm in its own fresh scratch repo: with (hooks + observer + critic, council mode + a raised done-gate), without (bare repo), and naive (bare repo, but one generic self-review sentence appended to the session's system prompt — the control isolating verified review vs. the agent nagging itself). Scoring (evals/ab/score.py) is entirely mechanical, never model-graded: hidden acceptance tests the agent never saw (tasks.py, CHECK-line partial credit), transcript facts (did a test command actually run), and git facts. The safety tier (safety_tasks.py) is the thesis measured head-on: realistic tickets with the safety requirement left implicit, scored by executing an adversarial exploit against the produced code (exit code, not text); each task carries good/bad reference impls so --selftest proves every scorer discriminates before anyone trusts a live run. Arm isolation is contamination-proof via the claude CLI's --setting-sources project,local (a lesson credited to the ponytail benchmark, whose baseline secretly ran the treatment through user-level hooks). --repo-url URL@sha optionally swaps the synthetic seed for a pinned real OSS repo (feature tier only; the shipped tasks' hidden tests then no longer apply as-is). Runs land as results.ndjsonl + report.md per run dir; published runs and methodology live in docs/benchmarks/.

Conventions

  • Tests are stdlib unittest, one file per concern in tests/ (the critic has several: beat/scheduler, receipts, council); anything touching model calls stubs the model via CRITIC_CMD (stubs answering multiple prompt kinds branch on prompt-content markers like TASK: REWRITE). The transcript fixture (tests/fixtures/session.jsonl) is synthetic.
  • Bounded-growth reads are role-dependent: hot paths and recency consumers use core.store.read_tail_rows; dedup sets and metric consumers (graded_ids, reflector/report.py, eval scoring) must stay unbounded — completeness matters there.
  • All NDJSON readers must tolerate a partial trailing line (files are appended mid-write) and skip unparseable lines rather than crash. observations.ndjsonl grows unbounded over a session, so whole-file consumers use core.store.read_tail_rows (and the UI server's readNdjsonTail) — the critic hot path is already O(new bytes) via byte offsets. Only reach back a bounded window; never re-parse the whole log per cycle.
  • Daemons never die on missing inputs — they wait; state files that fail to parse are discarded and rebuilt, not fatal.
  • Long text going into events/prompts is always truncated with an explicit … [N chars total] marker; caps are module-level constants.
  • State/ledger files a crash mid-write could corrupt are written with core.store.write_json_atomic, never a naked write_text.
  • Never write this repo's .codecouncil/ — it's live runtime data (the hooks are installed here, so the Critic reviews your session as you work; fix or COUNCIL-REBUTTAL: its findings, don't ignore them). Tests use temp dirs.