Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .claude/skills/verify/INSTRUCTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,16 @@ Run in parallel:
| Format | `ruff format --check mempalace_code/ tests/ scripts/` | 30s |
| Tests | `python -m pytest tests/ -x -q -m "not needs_network"` | 120s |
| Typecheck | `python -m pyright --pythonpath "$(python -c 'import sys; print(sys.executable)')"` | 120s |
| Strict slice typecheck | `python -m pyright -p pyrightconfig.strict.json` | 60s |
| Public safety | `python scripts/public_safety_scan.py --tracked --staged` | 30s |
| Scorecard | `python scripts/quality_scorecard.py --check` | 30s |

The scorecard check is stdlib-only (no install, no network) and validates the
quality scorecard's shape, determinism, and public-safety. It fails on malformed
or unsafe output. After a quality change lands, regenerate the committed
artifacts with `python scripts/quality_scorecard.py --write` (see
`docs/quality/README.md`).
quality scorecard's shape, determinism, public-safety, and committed artifact
freshness. The public-safety scan checks tracked and staged repository files for
private local paths, secret-like tokens, and local-only raw artifacts. After a
quality change lands, regenerate the committed artifacts with
`python scripts/quality_scorecard.py --write` (see `docs/quality/README.md`).

### If storage changed — add these

Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ jobs:
- run: pip install ruff
- run: ruff check mempalace_code/ tests/ scripts/
- run: ruff format --check mempalace_code/ tests/ scripts/
- name: Public-safety scan (tracked + staged)
run: python scripts/public_safety_scan.py --tracked --staged
- name: Quality scorecard (shape + determinism + public-safety)
run: python scripts/quality_scorecard.py --check

Expand All @@ -78,6 +80,8 @@ jobs:
- run: pip install -e ".[dev,chroma,spellcheck,treesitter]"
- name: Pyright typecheck
run: python -m pyright --pythonpath "$(python -c 'import sys; print(sys.executable)')"
- name: Pyright strict slice
run: python -m pyright -p pyrightconfig.strict.json

model-tests:
runs-on: ubuntu-latest
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ palace/
.venv/
venv/
.env
.verify-state

# IDE
.idea/
Expand Down
1 change: 0 additions & 1 deletion .verify-state

This file was deleted.

42 changes: 40 additions & 2 deletions docs/BACKLOG.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -80,26 +80,64 @@ items:
- CI or `/verify` runs the scorecard command and fails on malformed output.
resolution: "2026-06-05: Deterministic public scorecard: scripts/quality_scorecard.py emits Markdown+JSON; --check gates shape/determinism/public-safety in CI lint job and /verify; baseline committed to docs/quality/; update workflow in docs/quality/README.md"
done_summary: "Deterministic public scorecard: scripts/quality_scorecard.py emits Markdown+JSON; --check gates shape/determinism/public-safety in CI lint job and /verify; baseline committed to docs/quality/; update workflow in docs/quality/README.md"
- key: AUTOPILOT-DEMO-PUBLIC-SAFETY-GATE
section: autopilot_demo
status: done
priority: P1
summary: Add a repo-wide public-safety gate for tracked and staged files.
acceptance:
- Add a stdlib scanner that checks tracked worktree files and staged index blobs.
- Block secret-like tokens, real local machine paths, and local-only artifact paths without printing the matched secret text.
- Wire the scanner into CI and `/verify`.
- Remove or ignore any local-only artifact that is already tracked.
resolution: "2026-06-06: Added scripts/public_safety_scan.py --tracked --staged, CI and /verify wiring, focused tests, and removed tracked .verify-state while adding it to .gitignore."
done_summary: "Repo-wide public-safety scan now gates tracked/staged files and redacts matched content; .verify-state is local-only."
- key: AUTOPILOT-DEMO-RUFF-RATCHET
section: autopilot_demo
status: open
status: done
priority: P1
summary: Reduce transitional Ruff ignores in a measurable ratchet without broad style-only churn.
acceptance:
- Record baseline counts for global ignores, per-file ignores, and inline suppressions before edits.
- Remove at least one global ignore family or reduce per-file ignore entries by at least 25% for a focused module group.
- Replace any necessary inline suppressions with reasoned, narrow suppressions; no new blanket `noqa` entries.
- "`ruff check`, `ruff format --check`, Pyright, and the relevant focused tests pass; update the quality scorecard."
resolution: "2026-06-06: Reduced global Ruff ignores from 33 to 3 by keeping existing package/test debt scoped to per-file ignores and making new scripts inherit the stricter rule set."
done_summary: "Ruff global ignore ratchet: broad historical ignores are now scoped away from scripts; no new inline suppressions."
- key: AUTOPILOT-DEMO-PYRIGHT-STRICT-SLICE
section: autopilot_demo
status: open
status: done
priority: P1
summary: Establish a strict Pyright slice for stable low-level modules and expand it gradually.
acceptance:
- Add a strict typecheck config or command for a small stable slice such as config, reader, MCP tool profiles, and mining scanner.
- Replace ad hoc `Any` and broad dictionaries in that slice with typed contracts where the code owns the boundary.
- Keep compatibility shims and third-party stub gaps outside the first strict slice unless they can be fixed narrowly.
- CI or `/verify` runs both the existing basic Pyright check and the strict-slice check; update the quality scorecard.
resolution: "2026-06-06: Added pyrightconfig.strict.json for version.py, mcp_tool_profiles.py, and disk_budget.py; annotated disk_budget.py enough to pass strict; wired strict slice into CI and /verify."
done_summary: "Initial strict Pyright slice gates three stable low-level modules in CI and /verify."
- key: AUTOPILOT-DEMO-PYRIGHT-STRICT-SLICE-EXPANSION
section: autopilot_demo
status: open
priority: P1
summary: Expand the strict Pyright slice to config, reader, and mining scanner after adding typed boundaries.
acceptance:
- Add `mempalace_code/config.py`, `mempalace_code/reader.py`, and `mempalace_code/mining/scanner.py` to `pyrightconfig.strict.json`.
- Introduce typed config payload aliases, reader result types, gitignore rule types, and scan filter generics instead of broad dict/list shapes.
- Keep the existing basic Pyright gate green throughout.
- Update the quality scorecard and report the strict-slice expansion publicly.
- key: AUTOPILOT-DEMO-WORKFLOW-REVIEW-PROTOCOL
section: autopilot_demo
status: done
priority: P1
summary: Document the public-safe adversarial Claude workflow protocol for repo quality work.
acceptance:
- Define review lenses, skeptical refutation, synthesis, implementation, and verification steps.
- Separate publishable summaries from local-only raw workflow artifacts.
- Link the protocol from quality docs.
- Include the canonical quality verification commands.
resolution: "2026-06-06: Added docs/quality/workflow-review-protocol.md and linked it from docs/quality/README.md."
done_summary: "Public-safe multi-agent workflow review protocol is documented for future quality improvements."
- key: AUTOPILOT-DEMO-ARCHITECTURE-GUARD
section: autopilot_demo
status: open
Expand Down
19 changes: 14 additions & 5 deletions docs/quality/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ before/after evidence instead of becoming invisible churn.
|------|---------|
| [`scorecard.md`](scorecard.md) | Human-readable baseline (committed). |
| [`scorecard.json`](scorecard.json) | Machine-readable baseline (committed). |
| [`workflow-review-protocol.md`](workflow-review-protocol.md) | Public-safe multi-agent review protocol for quality work. |

Both are generated by [`scripts/quality_scorecard.py`](../../scripts/quality_scorecard.py).

Expand All @@ -24,8 +25,11 @@ python scripts/quality_scorecard.py --format both # both
# Regenerate the committed artifacts under docs/quality/
python scripts/quality_scorecard.py --write

# Validate shape, determinism, and public-safety (CI / verify gate)
# Validate shape, determinism, public-safety, and committed freshness (CI / verify gate)
python scripts/quality_scorecard.py --check

# Scan tracked/staged repository files for publish-safety leaks
python scripts/public_safety_scan.py --tracked --staged
```

The script uses **only the Python standard library** (no project import, no
Expand Down Expand Up @@ -54,11 +58,12 @@ absolute paths, no machine identifiers:
## Determinism & public-safety

`--check` builds the scorecard twice and fails if the JSON differs, validates
the output shape (types, required keys, sort order, invariants), and runs a
the output shape (types, required keys, sort order, invariants), checks that the
committed Markdown/JSON artifacts are fresh, and runs a rendered-output
public-safety scan for private paths and secret-like tokens. CI (`lint` job) and
`/verify` run `--check` so malformed or unsafe output fails the build. Because
`--check` validates *shape and determinism* (not exact counts), normal code
growth does not break CI — only a malformed scorecard does.
`/verify` also run `scripts/public_safety_scan.py --tracked --staged`, which
checks the repository files themselves for real local paths, secret-like tokens,
and local-only artifact paths.

## How Autopilot demo tasks update this scorecard

Expand All @@ -79,3 +84,7 @@ delta here:

Run `python scripts/quality_scorecard.py --check` before committing so the
regenerated artifacts stay well-formed and public-safe.

For adversarial multi-agent review of a quality change, use
[`workflow-review-protocol.md`](workflow-review-protocol.md) and publish only
the sanitized synthesis and verification evidence.
54 changes: 16 additions & 38 deletions docs/quality/scorecard.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
"code_size": {
"package_code_lines": 18978,
"package_files": 73,
"package_total_lines": 23441,
"test_files": 53,
"test_total_lines": 39096
"package_total_lines": 23442,
"test_files": 54,
"test_total_lines": 39198
},
"largest_modules": [
{
Expand Down Expand Up @@ -59,41 +59,11 @@
},
"ruff": {
"global_ignore_rules": [
"ARG001",
"ARG002",
"ARG005",
"B007",
"B027",
"B033",
"B904",
"B905",
"DTZ001",
"DTZ005",
"DTZ011",
"E501",
"PT001",
"PT006",
"PT017",
"RET504",
"RET505",
"RET508",
"SIM102",
"SIM103",
"SIM105",
"SIM108",
"SIM114",
"SIM115",
"SIM116",
"SIM117",
"UP006",
"UP015",
"UP024",
"UP028",
"UP035",
"UP036",
"UP045"
"SIM105"
],
"global_ignores": 33,
"global_ignores": 3,
"per_file_ignores": {
"by_pattern": [
{
Expand Down Expand Up @@ -178,13 +148,13 @@
"mempalace_code",
"tests"
],
"type_pyright_total": 108,
"type_pyright_total": 110,
"type_pyright_unreasoned": 0,
"unreasoned_total": 0
},
"tests": {
"test_files": 53,
"test_functions": 2259
"test_files": 54,
"test_functions": 2266
},
"verification_commands": [
{
Expand All @@ -203,6 +173,14 @@
"command": "python -m pyright --pythonpath \"$(python -c 'import sys; print(sys.executable)')\"",
"name": "typecheck"
},
{
"command": "python -m pyright -p pyrightconfig.strict.json",
"name": "typecheck_strict_slice"
},
{
"command": "python scripts/public_safety_scan.py --tracked --staged",
"name": "public_safety"
},
{
"command": "python scripts/quality_scorecard.py --check",
"name": "scorecard"
Expand Down
16 changes: 9 additions & 7 deletions docs/quality/scorecard.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ Schema version: 1
| Metric | Value |
|--------|------:|
| Package files (`mempalace_code/`) | 73 |
| Package total lines | 23441 |
| Package total lines | 23442 |
| Package code lines | 18978 |
| Test files (`tests/`) | 53 |
| Test total lines | 39096 |
| Test files (`tests/`) | 54 |
| Test total lines | 39198 |

## Largest Modules (top 10)

Expand All @@ -34,7 +34,7 @@ Schema version: 1
| Metric | Value |
|--------|------:|
| Selected rule families | 12 |
| Global ignores | 33 |
| Global ignores | 3 |
| Per-file ignore patterns | 2 |
| Per-file ignore entries | 38 |

Expand All @@ -58,7 +58,7 @@ Scope: `mempalace_code/`, `tests/` (excludes `tests/fixtures/`).

| Metric | Value |
|--------|------:|
| type/pyright ignores (total) | 108 |
| type/pyright ignores (total) | 110 |
| type/pyright unreasoned | 0 |
| noqa (total) | 41 |
| noqa blanket | 0 |
Expand All @@ -68,8 +68,8 @@ Scope: `mempalace_code/`, `tests/` (excludes `tests/fixtures/`).

| Metric | Value |
|--------|------:|
| Test files | 53 |
| Test functions | 2259 |
| Test files | 54 |
| Test functions | 2266 |

## Available Suites

Expand All @@ -91,4 +91,6 @@ Scope: `mempalace_code/`, `tests/` (excludes `tests/fixtures/`).
- **format**: `ruff format --check mempalace_code/ tests/ scripts/`
- **tests**: `python -m pytest tests/ -x -q -m "not needs_network"`
- **typecheck**: `python -m pyright --pythonpath "$(python -c 'import sys; print(sys.executable)')"`
- **typecheck_strict_slice**: `python -m pyright -p pyrightconfig.strict.json`
- **public_safety**: `python scripts/public_safety_scan.py --tracked --staged`
- **scorecard**: `python scripts/quality_scorecard.py --check`
72 changes: 72 additions & 0 deletions docs/quality/workflow-review-protocol.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Workflow Review Protocol

Public-safe protocol for using a multi-agent Claude workflow to improve this
repo without publishing raw local evidence.

## Trigger

Use this after a focused implementation is ready for review, especially for
quality-gate, release, security, dependency, CI, or public-demo changes.

## Inputs

- Current branch diff against `main`.
- Relevant public files only: source, tests, public docs, CI config, package
metadata.
- Explicit task acceptance criteria and verification commands.
- Local-only evidence paths may be used by the operator, but must not be pasted
into public artifacts.

## Lenses

Run independent reviewers for these lenses:

- Correctness: behavior, edge cases, failure modes.
- Determinism: stable output, ordering, caches, generated artifacts.
- Public-safety: secrets, private paths, local-only artifacts, publishable docs.
- Test coverage: real regression protection, not only happy-path assertions.
- Spec compliance: task acceptance criteria, repo rules, release boundaries.
- Maintainability: scoped design, duplication, future ratchets.

## Refutation

Each finding must pass a skeptical refutation step before implementation:

- Check the exact code path and tests.
- Reject findings based only on style preference or imagined behavior.
- Keep only findings with a concrete file, behavior, or missing gate.
- Record deliberate deferrals separately from fixes.

## Synthesis

The synthesis lead deduplicates surviving findings into:

- Implement now: actionable, low-ambiguity fixes in scope.
- Defer: valuable but separate backlog item.
- Reject: disproven or out-of-scope findings.

The public summary may include counts and categories. Do not publish raw model
transcripts, private local paths, hostnames, tokens, or task workspace contents.

## Implementation

Apply only vetted, scoped fixes. Update tests and canonical gates first, then
update public docs/backlog/scorecard. Keep raw workflow outputs under ignored
local paths such as `.tasks/`, `.protocols/`, or `docs/audits/`.

## Verification

Minimum verification after acting on a workflow review:

```bash
ruff check mempalace_code/ tests/ scripts/
ruff format --check mempalace_code/ tests/ scripts/
python -m pyright --pythonpath "$(python -c 'import sys; print(sys.executable)')"
python -m pyright -p pyrightconfig.strict.json
python scripts/public_safety_scan.py --tracked --staged
python scripts/quality_scorecard.py --check
python -m pytest tests/ -x -q -m "not needs_network"
```

If hosted workflow behavior matters, verify the real GitHub Actions run before
calling the change published.
Loading