Add deterministic Autopilot demo quality scorecard - #7
Merged
Conversation
…ity scorecard Add scripts/quality_scorecard.py — a stdlib-only, deterministic, public-safe scorecard that emits Markdown + JSON metrics for code size, largest modules, Ruff ignores, Pyright mode, unreasoned suppressions, test count, available CLI/MCP/smoke suites, and the canonical verification commands. Establishes the measurable baseline for the AUTOPILOT DEMO backlog section so future cleanup tasks report before/after deltas instead of inventing their own reporting. - One-command usage: `python scripts/quality_scorecard.py [--format both|json]`. - `--write` regenerates committed baseline under docs/quality/. - `--check` validates shape, self-determinism, and public-safety (no private paths/secrets); wired into the CI lint job and /verify, failing on malformed or unsafe output. No network, no project import — runs without install. - Suppression policy mirrors tests/test_type_suppressions.py; fixtures excluded. - Focused tests cover metric logic (hermetic synthetic repo), output shape, determinism, public-safety, validation, and entry points. Committed public-safe scorecard: docs/quality/scorecard.md, docs/quality/scorecard.json Update workflow for demo tasks: docs/quality/README.md Verification: ruff check mempalace_code/ tests/ scripts/ -> clean ruff format --check mempalace_code/ tests/ scripts/ -> clean python -m pyright -> 0 errors python scripts/quality_scorecard.py --check -> OK python -m pytest tests/test_quality_scorecard.py -q -> 27 passed python -m pytest tests/ -q -> 2313 passed Constraint: scorecard stays stdlib-only so it runs in the CI lint job and /verify without installing the package. Scope-risk: --check validates shape/determinism, not exact counts, so normal code growth must not break CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ial review Multi-agent review (6 lenses, each finding adversarially verified, then synthesized) surfaced real bugs and gaps in the baseline scorecard. Fixes, all stdlib-only / deterministic / public-safe / bounded: Correctness & determinism - Suppression scan is now tokenize-aware: only real comment tokens count, so a `# type: ignore`/`# noqa` mentioned inside a string/docstring is no longer miscounted (type_pyright_total 109 -> 108; headline unreasoned stays 0). - Test-def regex fallback tolerates PEP 695 generics (`def test_x[T](...)`) so the 3.11 SyntaxError fallback agrees with the 3.12+ AST count — the value is identical across machines. - File traversal skips build/dist/.egg-info/.venv/__pycache__/vendor noise so a stray local artifact cannot perturb the byte-identical output. Public-safety - The default stdout path now runs the same public-safety scan as --write and --check (both md and js, regardless of --format) and refuses to emit a leak. - --write no longer crashes on an absolute --out-dir outside the repo. - Forbidden-pattern set anchors /home/ on the bare prefix and adds the home/temp /Windows roots that user-controlled config fields could carry verbatim. Drift guards & tests - verification_commands typecheck entry aligned to the canonical /verify form; a new test asserts every command appears verbatim in verify/INSTRUCTIONS.md. - New test asserts committed docs/quality/* stay fresh vs a live render. - New test asserts the suppression regexes match tests/test_type_suppressions.py. - Added failure-path coverage for run_check, parametrized validate() negatives, non-UTF-8 fail-loud, --format both, json-arm refusal, and the tokenize fallback. Verification: ruff check mempalace_code/ tests/ scripts/ -> clean ruff format --check mempalace_code/ tests/ scripts/ -> clean python -m pyright -> 0 errors python scripts/quality_scorecard.py --check -> OK python -m pytest tests/ -q -> 2340 passed Deferred (out of scope for this bounded pass, see synthesis): tokenize-guarding the live test_type_suppressions.py gate; AWS/Slack/email/IPv4 forbidden patterns (no verbatim carrier in rendered output); a separate --check-committed CI mode (superseded by the freshness test). Constraint: scorecard stays stdlib-only; suppression scan must mirror the gate's policy. Scope-risk: explanatory comments/strings in the scorecard tests must avoid bare suppression syntax or they trip the raw-line gate and inflate the scan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a deterministic, public-safe quality scorecard for the Autopilot demo backlog. The scorecard emits Markdown and JSON, tracks repo-local quality signals, and adds CI/verify gates for shape, determinism, public-safety, and drift against committed artifacts.
Review follow-up applied before opening this PR:
git diff --check;Verification
ruff check mempalace_code/ tests/ scripts/ruff format --check mempalace_code/ tests/ scripts/python -m pyright --pythonpath "$(python -c 'import sys; print(sys.executable)')"\n-python scripts/quality_scorecard.py --check\n-python -m pytest tests/test_quality_scorecard.py -q\n-python -m pytest tests/ -q -m "not needs_network"->2342 passed, 2 deselected\n-git diff --check main..HEAD\n- repo public-safety scan for private paths/token-like literals: clean\n\n## Public-Safety\n\nNo.tasks/,.protocols/,docs/audits/, private benchmark outputs, private paths, or secret-like values are staged or committed.