Skip to content

chore(memory): memory-store baseline audit + checkpoint audit tooling - #1572

Open
aaj3f wants to merge 3 commits into
mainfrom
chore/memory-baseline-audit
Open

chore(memory): memory-store baseline audit + checkpoint audit tooling#1572
aaj3f wants to merge 3 commits into
mainfrom
chore/memory-baseline-audit

Conversation

@aaj3f

@aaj3f aaj3f commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

This started as the overdue housekeeping pass on .fluree-memory/repo.ttl and grew (deliberately, one PR — no stacking) into both halves of the problem: reconciling the store we have, and shipping the tooling so it never needs a heroic cleanup again.

Part 1 — the audited baseline

The committed store is what every contributor's agent loads for context-grounding, and it had drifted a long way from "durable repo knowledge": of 265 repo-scoped memories (244 in my working set + 21 added on main since), a large fraction were effort diaries — PR-stack checkpoints, "SHIPPED as #NNNN" notes, a deployed-stack snapshot with account IDs, session hand-off pointers, refs into scratch files and other people's home directories, four memories with literal serialized tool-call fragments corrupting their bodies. Roughly a quarter carried at least one artifactRef that no longer resolves at HEAD.

The part that made this urgent rather than cosmetic: recall is BM25 + re-rank, so every stale checkpoint dilutes ranking for memories that matter — and a confident memory asserting something the code no longer does is worse than no memory. The verification pass proved it: of ~60 nontrivial claims re-checked against the tree (agent-assisted, five parallel passes with exact-symbol evidence), more than twenty were materially wrong at HEAD — three "known gap" memories described gaps since closed (f:shapesSource, targetSubjectsOf indexing, f:schemaSource/f:rulesSource), the decimal-corruption memory predated the exactness fixes, two remembered env switches no longer exist, one flagged a CLI secret leak that #1534 already fixed.

Result: 265 → 214 memories. 147 kept, 59 rewritten in place to current-at-HEAD truth (same IDs, per the supersession model), 59 retired (11 folded into stronger survivors), 8 baseline memories added where the gap pass found uncaptured durable knowledge (grp_*.rs test aggregation, the binary-store gate on fast paths and why W3C-green never proves one fired, blank-node namespace conventions incl. the zero-arg BNODE() minting into the reserved fdb- namespace, the four parser depth guards, CI topology, policy fail-closed semantics and the real footgun — lenient IRI resolution silently narrowing enforcement scope — the fluree-db-consensus role, and the async-mutual-recursion trap). Everything retired stays recoverable via git log -p. The standard itself lives in docs/memory/guides/hygiene-and-auditing.md: five tests (durable-at-HEAD, team-audience, non-derivable, actionable, well-formed) + the repeatable procedure, written to be reusable verbatim on any repo using fluree memory.

Part 2 — checkpoint audit tooling, so the cleanup becomes routine

The insight from doing Part 1 by hand: the mechanical 80% of an audit (extract, flag narration/portability markers, check refs against HEAD, diff coverage) is exactly what the tool should own, while the judgment (the five tests) stays with the session. So:

  • memory_audit MCP tool + fluree memory audit CLI (fluree-db-memory/src/audit.rs, shared renderer in format.rs): scopes to the current branch's memories by default (branch provenance + refs intersecting the branch diff), flags narration/status language, over-cap content, absolute paths, missing refs, and churned refs (file modified since the memory was last verified, via git), plus a coverage section — files changed on the branch that no memory references, framed as "capture only what code/docs can't show." Read-only; degrades gracefully off-git; entries sort flagged-first so the display cap never hides findings.
  • mem:updatedAt re-verify stamps: every memory update (including a no-field-change call) stamps updatedAt — so "re-verified at HEAD" is one cheap call, recall's recency bonus uses last-verified rather than first-written, and churn signals measure something real. SCHEMA_VERSION_SALT bumped so existing ledgers rebuild to pick up the predicate. The reconciled baseline ships pre-stamped, since every memory in it was verified against origin/main today — without that, the first audit run flags ~93% of the store on churn alone.
  • Stale-ref markers in recall: results whose refs no longer exist at HEAD render with an explicit marker (text/json/context forms) — marked, not down-ranked, so the reading session is prompted to verify-and-update instead of the memory silently rotting.
  • The checkpoint habit, taught by the tool itself: the MCP server instructions and the shipped rules file now say to run memory_audit at PR-ready checkpoints — effort notes become authoritative or get retired there, not later. Since this ships in the crate, any repo adopting fluree memory gets the same hygiene loop for free.
  • CI guardrail: fluree-db-memory/tests/repo_memory_lint.rs lints the store itself in the ordinary workspace test job — parseability via the crate's own Turtle path, the 750-char cap, lowercase tags, repo-relative refs that must resolve at HEAD, no control bytes or tool-call debris, valid kinds/severity, and now the updatedAt shape.

Notes for review

Gates at HEAD of branch: fmt clean, clippy zero warnings on the touched crates, 96 tests green across fluree-db-memory/fluree-db-mcp (plus the CLI builds and was smoke-tested against this repo's real store — branch scoping, JSON output, and bad-base degradation all exercised). The repo.ttl diff is larger than the content change because the file regenerates in canonical write_memory_file order — best read per-block. One operational note: working trees that still have the old store will conflict on next pull; the MCP build-hash detection rebuilds cleanly from the merged file. Happy to walk through any block-level disposition — the retired set is all in git history if anyone wants to spot-check my judgment calls.

…onciled baseline

Reclassify all 265 repo-scoped memories (working set + main) against the
five-test rubric in docs/memory/guides/hygiene-and-auditing.md: 147 kept,
59 rewritten in place to current-at-HEAD truth (claims re-verified against
the tree), 59 retired (status snapshots, diff narration, superseded claims,
session state), 11 of the retirements folded into stronger survivors, and
8 baseline memories added (test-target aggregation, fast-path index gating,
blank-node namespaces, parser depth guards, CI topology, policy fail-closed
semantics, consensus crate role, async-recursion trap).

Guardrails: a CI lint of the store itself (fluree-db-memory/tests/
repo_memory_lint.rs — parseability, content cap, resolvable repo-relative
refs, no control bytes or tool-call debris, valid kinds/severity/tags), a
'What NOT to store' section in the shipped agent rules file, and the
hygiene guide documenting the editorial standard and audit procedure.

Retired memories remain recoverable via git history per the documented
supersession model.
@aaj3f
aaj3f requested a review from bplatz July 30, 2026 21:36
aaj3f added 2 commits July 30, 2026 18:19
…edAt re-verify stamps, stale-ref recall markers

Adds the mechanical half of the hygiene rubric so drift is visible at the
moment it matters — when an effort branch is about to be reviewed or merged.

- `audit.rs`: per-memory signals (over-cap content, progress/status narration,
  unportable paths and handles, tag shape, refs that no longer resolve, refs
  whose files have commits newer than the memory) plus a branch-coverage pass
  listing changed files no memory points at. Read-only; git failures degrade
  the coverage section to None rather than failing the audit.
- `memory_audit` MCP tool and `fluree memory audit [--all] [--base] [--format]`,
  both rendering the same markdown: the five tests up front, flagged memories
  first, output bounded.
- `mem:updatedAt` on every update — including one that changes no field, which
  is how an agent records "re-verified at HEAD". Recall's recency bonus and the
  audit's churn check both read it, so re-verification actually clears a flag.
- Recall now marks refs that no longer resolve in all three output forms.
Every memory in the reconciled store was audited against origin/main today;
updatedAt reflects that so audit churn signals start from a true baseline
instead of flagging the entire store on first run.
@aaj3f aaj3f changed the title chore(memory): rubric-based audit + reconciled baseline for the committed memory store chore(memory): memory-store baseline audit + checkpoint audit tooling Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant