Add deterministic Autopilot demo quality scorecard

.claude/skills/verify/INSTRUCTIONS.md

@@ -49,6 +49,13 @@ Run in parallel:
 | Format | `ruff format --check mempalace_code/ tests/ scripts/` | 30s |
 | Tests | `python -m pytest tests/ -x -q -m "not needs_network"` | 120s |
 | Typecheck | `python -m pyright --pythonpath "$(python -c 'import sys; print(sys.executable)')"` | 120s |
+| Scorecard | `python scripts/quality_scorecard.py --check` | 30s |
+
+The scorecard check is stdlib-only (no install, no network) and validates the
+quality scorecard's shape, determinism, and public-safety. It fails on malformed
+or unsafe output. After a quality change lands, regenerate the committed
+artifacts with `python scripts/quality_scorecard.py --write` (see
+`docs/quality/README.md`).
 
 ### If storage changed — add these
 

.github/workflows/ci.yml

@@ -59,8 +59,10 @@ jobs:
             pyproject.toml
             uv.lock
       - run: pip install ruff
-      - run: ruff check mempalace_code/ tests/
-      - run: ruff format --check mempalace_code/ tests/
+      - run: ruff check mempalace_code/ tests/ scripts/
+      - run: ruff format --check mempalace_code/ tests/ scripts/
+      - name: Quality scorecard (shape + determinism + public-safety)
+        run: python scripts/quality_scorecard.py --check
 
   typecheck:
     runs-on: ubuntu-latest

docs/BACKLOG.yaml

@@ -57,7 +57,6 @@ items:
       - Assert `mempalace-code search` exits successfully without `HF_TOKEN`, `unauthenticated`, `huggingface.co`, or retry noise on stdout/stderr.
       - Assert repeated `mempalace-code fetch-model` on a cached model uses local-only resolution and does not perform Hub metadata requests.
       - Keep the existing network-marked download tests separate from this no-network regression guard.
-
   - key: DEPENDENCY-SECURITY-UPGRADE-GATE
     section: quality
     status: open
@@ -69,18 +68,18 @@ items:
       - Run pip-audit or an equivalent resolver-level audit on fresh environments for the default install and every optional extra whose bounds change.
       - Keep deprecated optional backends capped away from affected ranges; do not raise ChromaDB to 1.x while GHSA-f4j7-r4q5-qw2c affects the available 1.x line.
       - Update uv.lock only after the audited resolver passes, then run hosted-CI-equivalent tests in a clean pip environment so stale local locks cannot hide dependency drift.
-
   - key: AUTOPILOT-DEMO-QUALITY-SCORECARD
     section: autopilot_demo
-    status: open
+    status: done
     priority: P1
     summary: Add a deterministic public quality scorecard that makes Autopilot cleanup progress visible across releases.
     acceptance:
       - Add a script or command that emits markdown and JSON metrics for code size, largest modules, Ruff ignore counts, Pyright mode, unreasoned suppressions, test count, and available smoke suites.
       - Generate any raw baseline evidence under local ignored docs/audits/, then commit only a sanitized public summary with public repo data and relative paths.
       - Document how each Autopilot demo task updates the scorecard with before/after metrics.
       - CI or `/verify` runs the scorecard command and fails on malformed output.
-
+    resolution: "2026-06-05: Deterministic public scorecard: scripts/quality_scorecard.py emits Markdown+JSON; --check gates shape/determinism/public-safety in CI lint job and /verify; baseline committed to docs/quality/; update workflow in docs/quality/README.md"
+    done_summary: "Deterministic public scorecard: scripts/quality_scorecard.py emits Markdown+JSON; --check gates shape/determinism/public-safety in CI lint job and /verify; baseline committed to docs/quality/; update workflow in docs/quality/README.md"
   - key: AUTOPILOT-DEMO-RUFF-RATCHET
     section: autopilot_demo
     status: open
@@ -91,7 +90,6 @@ items:
       - Remove at least one global ignore family or reduce per-file ignore entries by at least 25% for a focused module group.
       - Replace any necessary inline suppressions with reasoned, narrow suppressions; no new blanket `noqa` entries.
       - "`ruff check`, `ruff format --check`, Pyright, and the relevant focused tests pass; update the quality scorecard."
-
   - key: AUTOPILOT-DEMO-PYRIGHT-STRICT-SLICE
     section: autopilot_demo
     status: open
@@ -102,7 +100,6 @@ items:
       - Replace ad hoc `Any` and broad dictionaries in that slice with typed contracts where the code owns the boundary.
       - Keep compatibility shims and third-party stub gaps outside the first strict slice unless they can be fixed narrowly.
       - CI or `/verify` runs both the existing basic Pyright check and the strict-slice check; update the quality scorecard.
-
   - key: AUTOPILOT-DEMO-ARCHITECTURE-GUARD
     section: autopilot_demo
     status: open
@@ -113,7 +110,6 @@ items:
       - Enforce at minimum that storage/config/mining core does not import CLI, MCP dispatch, or Chroma-only modules.
       - Report cycles and boundary violations with file-level paths that are easy for Autopilot to fix.
       - Add tests for the guard itself and wire the command into `/verify` or CI; update the quality scorecard.
-
   - key: AUTOPILOT-DEMO-CLI-GOLDEN-SCENARIOS
     section: autopilot_demo
     status: open
@@ -124,7 +120,6 @@ items:
       - Cover at least one important guard/failure path such as missing palace, invalid read range, or unsafe mirror preflight.
       - Force offline/version-check-disabled environment variables so the suite is deterministic and network-free.
       - Remove all smoke artifacts or report them explicitly; update release/verify docs and the quality scorecard.
-
   - key: AUTOPILOT-DEMO-MCP-STDIO-CONTRACTS
     section: autopilot_demo
     status: open
@@ -135,7 +130,6 @@ items:
       - Verify `minimal`, `code`, `kg`, `notes`, and `full` profiles expose only the expected tools.
       - Include representative success and error responses for read-only, write, graph/KG, and disabled-tool paths.
       - Keep direct handler tests as fast unit coverage but label them separately from real MCP stdio coverage; update the quality scorecard.
-
   - key: AUTOPILOT-DEMO-SECURITY-BOUNDARY-TESTS
     section: autopilot_demo
     status: open
@@ -146,7 +140,6 @@ items:
       - Prefer standard-library table-driven tests; add a fuzz/property-test dependency only if it clearly improves signal and stays dev-only.
       - Every rejected input returns a stable error code/message and does not create partial palace state.
       - Document any remaining accepted risk in a public-safe plan or backlog follow-up; update the quality scorecard.
-
   - key: AUTOPILOT-DEMO-PERF-BUDGETS
     section: autopilot_demo
     status: open
@@ -157,7 +150,6 @@ items:
       - Measure mine time, incremental no-op time, search latency, read latency, and cleanup/optimize smoke duration.
       - Store conservative budgets and machine-independent comparison rules so CI failures indicate meaningful regressions, not normal hardware variance.
       - Publish before/after numbers in the quality scorecard and release notes when a demo task improves a budget.
-
   - key: AUTOPILOT-DEMO-DOCS-DRIFT-GUARD
     section: autopilot_demo
     status: open

docs/plans/AUTOPILOT-DEMO-QUALITY-ROADMAP.md

@@ -41,9 +41,15 @@ destabilizing it. Each task should produce three things:
 
 ## Suggested Sequence
 
-1. `AUTOPILOT-DEMO-QUALITY-SCORECARD`
+1. `AUTOPILOT-DEMO-QUALITY-SCORECARD` — **done (baseline established)**
    - Establish the baseline first. Later tasks should update it instead of
      inventing their own reporting format.
+   - Implemented as `scripts/quality_scorecard.py` (stdlib-only, deterministic,
+     public-safe). Emits Markdown + JSON; `--write` regenerates the committed
+     baseline under `docs/quality/`, `--check` gates shape/determinism/public-
+     safety in CI (`lint` job) and `/verify`.
+   - Baseline artifacts: `docs/quality/scorecard.md`, `docs/quality/scorecard.json`.
+   - Update workflow for later tasks: `docs/quality/README.md`.
 
 2. `AUTOPILOT-DEMO-CLI-GOLDEN-SCENARIOS` and `AUTOPILOT-DEMO-MCP-STDIO-CONTRACTS`
    - These make the demo credible because they prove real user surfaces before
@@ -76,3 +82,7 @@ destabilizing it. Each task should produce three things:
 
 These are not criticisms by themselves. They are the useful visible surfaces for
 showing controlled, evidence-backed quality improvement.
+
+The current measured values for these signals live in the committed scorecard
+(`docs/quality/scorecard.json`); regenerate with
+`python scripts/quality_scorecard.py --write`.

docs/quality/README.md

@@ -0,0 +1,81 @@
+# Quality Scorecard
+
+A deterministic, public-safe snapshot of code-quality signals for the
+`AUTOPILOT DEMO` backlog section. It exists so cleanup work has measurable
+before/after evidence instead of becoming invisible churn.
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| [`scorecard.md`](scorecard.md) | Human-readable baseline (committed). |
+| [`scorecard.json`](scorecard.json) | Machine-readable baseline (committed). |
+
+Both are generated by [`scripts/quality_scorecard.py`](../../scripts/quality_scorecard.py).
+
+## Generate
+
+```bash
+# Print to stdout
+python scripts/quality_scorecard.py                 # Markdown
+python scripts/quality_scorecard.py --format json   # JSON
+python scripts/quality_scorecard.py --format both   # both
+
+# Regenerate the committed artifacts under docs/quality/
+python scripts/quality_scorecard.py --write
+
+# Validate shape, determinism, and public-safety (CI / verify gate)
+python scripts/quality_scorecard.py --check
+```
+
+The script uses **only the Python standard library** (no project import, no
+network), so it runs anywhere Python 3.11+ is available.
+
+## What it measures
+
+All metrics are repo-local and public-safe by construction — no timestamps, no
+absolute paths, no machine identifiers:
+
+- **Code size** — package/test file counts and line totals.
+- **Largest modules** — top modules by line count (relative paths only).
+- **Ruff ignores** — global ignore count, selected rule families, and per-file
+  ignore patterns/entries from `pyproject.toml`.
+- **Pyright** — type-checking mode and strict status.
+- **Suppressions** — total and *unreasoned* type/pyright ignores and blanket
+  `noqa`, using the same policy as
+  [`tests/test_type_suppressions.py`](../../tests/test_type_suppressions.py)
+  (`# type: ignore[code]  # reason: ...`). Fixture dirs are excluded.
+- **Tests** — test file and test-function counts.
+- **Suites** — which CLI/MCP/smoke surfaces exist.
+- **Verification commands** — the canonical `/verify` pre-commit checks (kept
+  verbatim-identical to `.claude/skills/verify/INSTRUCTIONS.md`, enforced by a
+  drift test).
+
+## Determinism & public-safety
+
+`--check` builds the scorecard twice and fails if the JSON differs, validates
+the output shape (types, required keys, sort order, invariants), and runs a
+public-safety scan for private paths and secret-like tokens. CI (`lint` job) and
+`/verify` run `--check` so malformed or unsafe output fails the build. Because
+`--check` validates *shape and determinism* (not exact counts), normal code
+growth does not break CI — only a malformed scorecard does.
+
+## How Autopilot demo tasks update this scorecard
+
+Every task in the `AUTOPILOT DEMO` backlog section should record a before/after
+delta here:
+
+1. **Before** the change, capture the baseline: `python scripts/quality_scorecard.py --format json`.
+   Keep raw/private working notes only under the git-ignored `docs/audits/`;
+   never commit them.
+2. Make the focused change (e.g. remove a Ruff ignore family, add a strict
+   Pyright slice, add a real CLI/MCP suite).
+3. **After** the change, regenerate the committed artifacts:
+   `python scripts/quality_scorecard.py --write`.
+4. In the PR/commit description and release notes, cite the moved metric using
+   public repo data only — e.g. "unreasoned suppressions N → M",
+   "Ruff global ignores N → M", "Pyright strict slice added". The committed
+   `scorecard.json` diff is the evidence.
+
+Run `python scripts/quality_scorecard.py --check` before committing so the
+regenerated artifacts stay well-formed and public-safe.