feat(eeg): add fail-closed structured-state shadow artifacts

[aurascoper]

Prerequisite for #33. Commits the structured-state module that #33's
test_synthetic_structured_state_replay_remains_shadow_only imports — it had
only ever existed as an untracked working-tree file, so that test could not pass
on any checkout.

Scope

NeuralComposeEEG/src/neuralcompose_eeg/structured_state.py
NeuralComposeEEG/tests/test_structured_state.py

Nothing else. No schema document was needed — the module's nc-eeg-encoder-state-v0
shape is defined and enforced in code, and every dependency (ContractError,
CanonicalDataset, sha256_json) already exists on this base.

What it does

Serializes offline encoder probabilities into a replayable shadow-state record
plus a hash-bound manifest. Both carry the same status block, enforced on write
and re-verified on read:

status:            insufficient_evidence
science_status:    pipeline_only
shadow_only:       true
live_control:      false
promotion_status:  not_eligible

The reader re-derives confidence, entropy_nats, and argmax_observable from
the stored probabilities rather than trusting the written values, so a tampered
record is rejected even when its manifest hash has been recomputed to match it.
A forbidden-term check rejects any artifact whose serialized form mentions
dialogue, dialectic, prompt, speech, policy, action, waveform, label, or target.

Two changes made during audit

The module was audited as an independent component before staging, per the
acceptance contract. It was coherent and self-contained; two gaps were closed.

1. Publication is now atomic. write_text could leave a truncated artifact
over a previously valid one if interrupted mid-write. Both outputs now publish
through a same-directory rename:

def _publish_text(path: Path, text: str) -> None:
    partial = path.with_name(f".{path.name}.partial")
    try:
        partial.write_text(text, encoding="utf-8")
        os.replace(partial, path)
    finally:
        partial.unlink(missing_ok=True)

2. Eight tests added for invariants the module enforced but never proved:

test_rejects_probability_shape_mismatch
test_rejects_nonfinite_probabilities
test_records_dataset_and_encoder_provenance
test_replay_rejects_manifest_granting_live_control_or_promotion
test_replay_rejects_record_granting_live_control
test_replay_rejects_argmax_that_is_not_the_maximum
test_publication_leaves_no_partial_artifacts
test_rejected_republication_preserves_the_previous_artifact

The last one is the atomicity proof: a rejected republication leaves the prior
artifact byte-identical, leaves no .partial behind, and the prior artifact
still loads.

Acceptance contract status

deterministic artifact generation           covered
probability shape + sum-to-one validation   covered
non-finite probability rejection            covered
source/dataset/encoder provenance           covered
artifact + manifest SHA-256 validation      covered
atomic publication                          covered (added here)
shadow_only / live_control / promotion      covered (write and read)
no application-runtime imports              covered (numpy + stdlib only)
no model download or execution              covered
no dialogue/review data in artifacts        covered
backward-compatible loading                 n/a — v0 is the only schema

Open decision: republication overwrites silently

The contract also listed "no path escape or silent overwrite". Path escape does
not apply — the operator names both output paths explicitly on the CLI, so there
is no root to escape from.

Silent overwrite is a real, deliberate behaviour and is left as authored:
re-running the tool over existing artifacts replaces them without prompting.
test_round_trip_is_deterministic_and_shadow_only depends on this, writing twice
to the same paths and asserting byte-identical output.

That is idempotent republication, which seems intended. Making it refuse to
overwrite differing content would be a semantic change to the publication
contract, so it is flagged here rather than made unilaterally. Say the word and
it becomes a follow-up.

Verification

python -m unittest NeuralComposeEEG.tests.test_structured_state    13 passed
python -m unittest discover -s NeuralComposeEEG/tests -t .         46 passed
py_compile (module + test)                                         OK
git diff --check                                                   clean

Run with the pinned stack (numpy 1.26.4 / torch 2.4.1). The module itself needs
only numpy; torch enters through test_pipeline, whose fixture helpers this test
reuses.

Note that CI does not yet run any Python suite — build-and-test is Swift-only,
which is why the missing module was never caught. Wiring that up is a separate
small PR, to land before #33 goes ready.

[aurascoper]

Related: #37 puts NeuralComposeEEG/tests under CI for the first time, so this module's 13 contract tests become gating once both land. Until #37 merges, no Python suite runs in CI at all — which is why the missing module reached #33 unnoticed.

[aurascoper]

Review — two changes before merge

The module is coherent, scoped correctly, and the new numerical/status tests are valuable. I am holding the merge for two fail-closed contract gaps.

1. Replay does not yet re-verify the complete provenance contract

load_shadow_state_records() verifies the status block, record count/hash, probabilities, confidence, entropy, and argmax. It does not validate that:

• manifest["encoder"] itself satisfies _validate_encoder_provenance;
• every record's encoder equals the manifest encoder;
• every record's source.dataset_sha256 and source.source_manifest_sha256 equal the manifest values;
• state_id is hexadecimal and recomputes from the declared dataset/window hashes.

Today a record can have its encoder/source provenance changed, the manifest's records_sha256 recomputed, and replay will accept an internally contradictory artifact. Please add focused rejection tests and enforce those equalities on read.

Suggested tests:

• test_replay_rejects_record_encoder_mismatch_with_manifest
• test_replay_rejects_record_source_mismatch_with_manifest
• test_replay_rejects_invalid_or_unrecomputed_state_id

2. Publication is atomic per file, not for the state/manifest pair

The state file is replaced before the manifest. A crash or manifest-write failure in between can replace a previously valid state file while leaving the old manifest, losing the prior valid pair. The current rejected-republication test fails during validation before either publish and does not exercise this seam.

Decision on the open overwrite question:

• Allow byte-identical idempotent republication.
• Refuse differing content by default. These are provenance artifacts, not mutable cache entries.
• Add an explicit --replace-existing later only if a real workflow requires it.

Minimal safe contract:

1. Precompute and validate both payloads before touching outputs.
2. Reject identical resolved output paths.
3. If both outputs exist and both bytes match, return as a no-op.
4. If either existing output differs, raise ContractError without changing either file.
5. For a first publication, write the state file and publish the manifest last as the commit marker.
6. Permit repair of an interrupted first publication only when the existing state bytes exactly match the newly computed state and the manifest is absent.

Please add a test that a valid but different republication is refused and preserves both prior files byte-for-byte, plus a test for the interrupted-first-publication repair path.

Once these pass with the full 46-test EEG suite, #35 is ready to merge. No change is requested to the shadow-only/runtime-authority boundary.

[aurascoper]

Both blockers addressed in bbeb979, with 5a72fd1 preserved un-amended and the current base merged in via 7d1493a (so this now runs against the non-vacuous python-contracts job from #37).

Blocker 1 — replay now binds records to their manifest

The loader was validating the two artifacts independently. It now requires agreement:

• manifest["encoder"] validated through _validate_encoder_provenance, and record["encoder"] must equal it
• each record's dataset_sha256 and source_manifest_sha256 must equal the manifest's
• raw_window_sha256 validated as a lowercase hex digest
• state_id recomputed from dataset_sha256 + raw_window_sha256 rather than accepted as any unique 64-character string

state_id derivation moved into a shared _state_id() helper so the writer and the replay check cannot drift apart.

Blocker 2 — publication commits the pair

Both payloads are built and validated before anything is written; the manifest is written last as the commit marker. The overwrite policy is exactly as specified:

case	behaviour
byte-identical	no-op, nothing written
any difference	refused, both files untouched
records-only, manifest absent	completable — interrupted-publication repair, only when records match exactly
same path for both outputs	rejected outright

Explicit replacement deliberately not added.

The guard immediately caught a live bug

On first run, 17 of 20 tests failed with refusing to overwrite existing state manifest. The cause: these tests published their state manifest to root/manifest.json — the same path test_pipeline._manifest writes the source manifest to. Every run had been silently clobbering the fixture, and the old per-file overwrite hid it completely. Artifacts now publish under root/shadow/.

Acceptance

structured_state_tests:  20 passed   (was 13; +7 from this review)
NeuralComposeEEG_tests:  53 passed, 0 failures
eval_contracts:          59 passed, 3 deselected
py_compile:              OK
git_diff_check:          clean

One deviation from the stated criteria worth flagging: the EEG suite is 53, not the expected 46. 46 assumed the pre-review 13 structured-state tests; the six required cases plus a raw_window_sha256 digest-validation case make it 20, so 33 + 20 = 53. CI counts pending.

[aurascoper]

Merged after independent review of the repaired head bbeb979.

Final acceptance:

• record/manifest encoder and source provenance are bound on replay;
• raw_window_sha256 and state_id are validated as lowercase SHA-256 digests, and state_id is re-derived from dataset + raw-window identity;
• both payloads are constructed and checked before publication;
• byte-identical republication is a no-op;
• differing republication is refused without changing either artifact;
• records-only interrupted publication can complete only when the records match exactly;
• manifest publishes last as the pair's commit marker;
• same output path is rejected;
• fixture outputs moved under shadow/, exposing and removing the prior source-manifest clobber.

Verification at the merged head:

• NeuralComposeEEG: 53 passed;
• Tests/eval: 62 collected, 3 explicitly deselected under issue Three pre-existing Tests/eval failures deselected by the new Python CI job #38, 59 passed;
• Swift build-and-test: passed;
• Python contracts job: passed;
• py_compile and git diff --check: clean.

Merged as 54080a682c5f928a26d7135ade691737ef53c5e0. The original implementation commit 5a72fd1 remains unamended; base merge 7d1493a and repair bbeb979 remain visible.