v0.6.0: Proactive provenance (run-record + node runner) + LaTeX paper standard

[hebbianloop]

Summary

Makes the Glimmer graph executable, not just descriptive, and establishes a LaTeX standard for submittable papers.

The agentic loop (docs/agentic-loop.md) was specified but had no runtime primitive; the only "verification" script re-hashed files on disk ("not a full re-run"). This PR closes that gap with one core node type and one tool, and lands two manuscripts.

Schema (v0.5.1 → v0.6.0)

• run-record core node type — one concrete, replayable invocation (PROV Activity) with a planned → ready → running → executed lifecycle, binding a method + pinned, standard-validated inputs + expected outputs + command + container digest + a runner-written verdict. The executable unit of the agentic loop.
• New edges reruns / consumes / regenerates / emits (+ inverse regenerated-by); tests-hypothesis / addresses-concept allowed from a run-record.
• Method-registry affordances: registry-ref + implements / equivalent-to / refines. standard.validator hint for the runtime gate.

Runner — glimmer/tools/run.py (glimmer run / glimmer rerun)

• Pre-run standards gate (inputs pinned AND valid; delegated to each standard's validator).
• datalad containers-run replay (feature-detected; degrades honestly).
• Three verification tiers: byte-identical (NIfTI/GIFTI/JSON normalization), numeric-within-tolerance (re-derive published numbers), structural (agent/LLM).
• certify_equivalence, dependency-ordered all, provenance manifest, verdict writeback.

Validator, example, docs

• validate.py: run-record field/edge/target/lifecycle checks + agent-protocol rule.
• examples/synthetic-provenance/: exercises the loop + all three tiers + the gate + equivalence, no real data / no heavy deps.
• New docs/proactive-provenance.md; agentic-loop.md made executable; roadmap renumbered (federation → v0.7, registry/minimal-path folded in); README repositioned as AI-native reproducibility; agent-protocol + datalad-pattern cross-linked.

Papers (new LaTeX standard)

• papers/glimmer-paper.cls — venue-neutral house class; papers/README.md documents the convention.
• Paper 1 (01-knowledge-graph-navigation/) — scoped to v0.5: the graph as a navigable substrate; verification as a validator-enforced contract, the executable runner named as forthcoming. Preprint candidate (9pp).
• Paper 2 (02-proactive-provenance/) — v0.6: the run-record + node runner (7pp). LaTeX port of docs/paper-draft.md.

Verification

• glimmer validate on the synthetic example: 0 errors. glimmer rerun … all → 100% (verified ×2, reproduced-within-tolerance, structurally-valid).
• Negative controls: gate-fail (exit 1), tamper → mismatch, offline → inputs-unavailable, equivalence certified by output-match.
• Backward-compat: ds000114-nipype validates unchanged.
• Both papers compile via make (latexmk + bibtex), no undefined references.

🤖 Generated with Claude Code

[hebbianloop]

Review — v0.6 proactive provenance (runner + run-record)

I reviewed the branch and ran the worked example end-to-end (emit_graph → validate → run → rerun --manifest → negative controls). It works: forward run executes all four, rerun gives verified (byte-identical) ×2, reproduced-within-tolerance (classifier), structurally-valid (agent), and gate-failed on the malformed-input negative control. The three-tier design, the delegated standards gate, the honest-degradation discipline (never a silent pass), and the importable public surface are all solid. This is the runner the whole effort needed — recommend merge after the items below.

High (correctness / claims)

1. datalad containers-run path is untested and likely misuses the API. execute() calls containers-run -n f"{image}@{digest}", but -n expects a registered container name, not image@digest. Every run-record in the worked example is a plain python3 ... command with no container-digest, so the headline result runs entirely via the host-subprocess fallback — the pinned-container replay path (the center of the reproducibility argument, and of Paper 02) is never exercised. Please add a container-backed example that actually drives containers-run, fix the invocation to register/resolve the container, and soften the paper claim until a container path is demonstrated.
2. Command splitting. The datalad paths use cmd.split() (whitespace), which breaks quoted args or paths with spaces; the host path uses shell=True. Inconsistent and fragile for real commands (FSL/Nipype with real paths). Use shlex.split() for the argv paths.
3. normalized_hash suffix detection. "".join(path.suffixes) misfires on filenames containing dots (e.g. sub-01.run-1.nii.gz → .run-1.nii.gz, no match → silent fallback to raw hash → false mismatch on volatile headers, defeating byte-identical verification). Use name.lower().endswith((".nii", ".nii.gz", ".gii", ".json")).

Medium

4. run_root = rokb.parent is a layout assumption. If the rokb dir is the project root (code/inputs/out as siblings of the index), run_root points one level too high and all relative paths break; the guard only catches filesystem root. Make it explicit or detect code/.
5. "reproducibility-rate-pct" semantics. Forward run marks everything executed → 100% regardless of reproduction; rerun all counts the deliberate gate-fail negative control against the rate (I observed 80%). The metric conflates execution, reproduction, and correct-negative behavior. Report separate counts and exclude negative controls; make the paper's "100%" precise (4/4 on the positive set).
6. Federation security (roadmap flag). execute()/validators run graph-supplied strings via shell=True. Fine for a local trusted graph, but the v0.7 cross-project/federation direction means pulling run-records from other projects — executing those is arbitrary code execution. Federated run-records need sandboxing / explicit approval; note it now so the governance layer accounts for it.

Low / polish

7. write_verdict_back reloads the whole graph per node, and the CLI reloads again after each writeback → O(N²) sidecar parses with --write-verdict. Cache paths.
8. Dead expression: pin if "output-path" in pin else {**pin, "output-path": pin.get("output-path")} (both branches equivalent).
9. validate warns "dataset has no produced-by edge" on the synthetic raw input datasets — suppress for declared-raw inputs (or mark them) so the example validates clean.
10. "contract" wording. The v0.6 docs lean on "executable contract / self-checking contract"; per the review on glimmer-web#19 we're dropping "contract" in prose. Mechanism stays; rename in proactive-provenance.md + schema.md.

Alignment

This is exactly the direction: registry-ref / equivalent-to / certify_equivalence are the seeds of the v0.7 recomputation-avoidance + federation layer. The integration seam with the ADS-side tooling is clear: the planner's runnable frontier (glimmer_sequence.py --frontier) should emit run-records that glimmer run executes. None of the above blocks merge; #1–#3 are the ones I'd fix before leaning on the reproducibility claim in the papers.