chore(memory): memory-store baseline audit + checkpoint audit tooling - #1572
Open
aaj3f wants to merge 3 commits into
Open
chore(memory): memory-store baseline audit + checkpoint audit tooling#1572aaj3f wants to merge 3 commits into
aaj3f wants to merge 3 commits into
Conversation
…onciled baseline Reclassify all 265 repo-scoped memories (working set + main) against the five-test rubric in docs/memory/guides/hygiene-and-auditing.md: 147 kept, 59 rewritten in place to current-at-HEAD truth (claims re-verified against the tree), 59 retired (status snapshots, diff narration, superseded claims, session state), 11 of the retirements folded into stronger survivors, and 8 baseline memories added (test-target aggregation, fast-path index gating, blank-node namespaces, parser depth guards, CI topology, policy fail-closed semantics, consensus crate role, async-recursion trap). Guardrails: a CI lint of the store itself (fluree-db-memory/tests/ repo_memory_lint.rs — parseability, content cap, resolvable repo-relative refs, no control bytes or tool-call debris, valid kinds/severity/tags), a 'What NOT to store' section in the shipped agent rules file, and the hygiene guide documenting the editorial standard and audit procedure. Retired memories remain recoverable via git history per the documented supersession model.
…edAt re-verify stamps, stale-ref recall markers Adds the mechanical half of the hygiene rubric so drift is visible at the moment it matters — when an effort branch is about to be reviewed or merged. - `audit.rs`: per-memory signals (over-cap content, progress/status narration, unportable paths and handles, tag shape, refs that no longer resolve, refs whose files have commits newer than the memory) plus a branch-coverage pass listing changed files no memory points at. Read-only; git failures degrade the coverage section to None rather than failing the audit. - `memory_audit` MCP tool and `fluree memory audit [--all] [--base] [--format]`, both rendering the same markdown: the five tests up front, flagged memories first, output bounded. - `mem:updatedAt` on every update — including one that changes no field, which is how an agent records "re-verified at HEAD". Recall's recency bonus and the audit's churn check both read it, so re-verification actually clears a flag. - Recall now marks refs that no longer resolve in all three output forms.
Every memory in the reconciled store was audited against origin/main today; updatedAt reflects that so audit churn signals start from a true baseline instead of flagging the entire store on first run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This started as the overdue housekeeping pass on
.fluree-memory/repo.ttland grew (deliberately, one PR — no stacking) into both halves of the problem: reconciling the store we have, and shipping the tooling so it never needs a heroic cleanup again.Part 1 — the audited baseline
The committed store is what every contributor's agent loads for context-grounding, and it had drifted a long way from "durable repo knowledge": of 265 repo-scoped memories (244 in my working set + 21 added on main since), a large fraction were effort diaries — PR-stack checkpoints, "SHIPPED as #NNNN" notes, a deployed-stack snapshot with account IDs, session hand-off pointers, refs into scratch files and other people's home directories, four memories with literal serialized tool-call fragments corrupting their bodies. Roughly a quarter carried at least one
artifactRefthat no longer resolves at HEAD.The part that made this urgent rather than cosmetic: recall is BM25 + re-rank, so every stale checkpoint dilutes ranking for memories that matter — and a confident memory asserting something the code no longer does is worse than no memory. The verification pass proved it: of ~60 nontrivial claims re-checked against the tree (agent-assisted, five parallel passes with exact-symbol evidence), more than twenty were materially wrong at HEAD — three "known gap" memories described gaps since closed (
f:shapesSource,targetSubjectsOfindexing,f:schemaSource/f:rulesSource), the decimal-corruption memory predated the exactness fixes, two remembered env switches no longer exist, one flagged a CLI secret leak that #1534 already fixed.Result: 265 → 214 memories. 147 kept, 59 rewritten in place to current-at-HEAD truth (same IDs, per the supersession model), 59 retired (11 folded into stronger survivors), 8 baseline memories added where the gap pass found uncaptured durable knowledge (
grp_*.rstest aggregation, the binary-store gate on fast paths and why W3C-green never proves one fired, blank-node namespace conventions incl. the zero-argBNODE()minting into the reservedfdb-namespace, the four parser depth guards, CI topology, policy fail-closed semantics and the real footgun — lenient IRI resolution silently narrowing enforcement scope — thefluree-db-consensusrole, and the async-mutual-recursion trap). Everything retired stays recoverable viagit log -p. The standard itself lives indocs/memory/guides/hygiene-and-auditing.md: five tests (durable-at-HEAD, team-audience, non-derivable, actionable, well-formed) + the repeatable procedure, written to be reusable verbatim on any repo usingfluree memory.Part 2 — checkpoint audit tooling, so the cleanup becomes routine
The insight from doing Part 1 by hand: the mechanical 80% of an audit (extract, flag narration/portability markers, check refs against HEAD, diff coverage) is exactly what the tool should own, while the judgment (the five tests) stays with the session. So:
memory_auditMCP tool +fluree memory auditCLI (fluree-db-memory/src/audit.rs, shared renderer informat.rs): scopes to the current branch's memories by default (branch provenance + refs intersecting the branch diff), flags narration/status language, over-cap content, absolute paths, missing refs, and churned refs (file modified since the memory was last verified, via git), plus a coverage section — files changed on the branch that no memory references, framed as "capture only what code/docs can't show." Read-only; degrades gracefully off-git; entries sort flagged-first so the display cap never hides findings.mem:updatedAtre-verify stamps: everymemory update(including a no-field-change call) stampsupdatedAt— so "re-verified at HEAD" is one cheap call, recall's recency bonus uses last-verified rather than first-written, and churn signals measure something real.SCHEMA_VERSION_SALTbumped so existing ledgers rebuild to pick up the predicate. The reconciled baseline ships pre-stamped, since every memory in it was verified against origin/main today — without that, the first audit run flags ~93% of the store on churn alone.memory_auditat PR-ready checkpoints — effort notes become authoritative or get retired there, not later. Since this ships in the crate, any repo adoptingfluree memorygets the same hygiene loop for free.fluree-db-memory/tests/repo_memory_lint.rslints the store itself in the ordinary workspace test job — parseability via the crate's own Turtle path, the 750-char cap, lowercase tags, repo-relative refs that must resolve at HEAD, no control bytes or tool-call debris, valid kinds/severity, and now theupdatedAtshape.Notes for review
Gates at HEAD of branch: fmt clean, clippy zero warnings on the touched crates, 96 tests green across
fluree-db-memory/fluree-db-mcp(plus the CLI builds and was smoke-tested against this repo's real store — branch scoping, JSON output, and bad-base degradation all exercised). Therepo.ttldiff is larger than the content change because the file regenerates in canonicalwrite_memory_fileorder — best read per-block. One operational note: working trees that still have the old store will conflict on next pull; the MCP build-hash detection rebuilds cleanly from the merged file. Happy to walk through any block-level disposition — the retired set is all in git history if anyone wants to spot-check my judgment calls.