diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index d45362e8..303474f8 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,11 @@ +## 2026-07-19 — cosine-free ingestion leg: 2-agent fleet (Sonnet grindwork + Opus filigree) — main-thread orchestration + +- **Task:** operator ("go ahead autoattended autonomous … opus agents for filigrane, sonnet 5 agents for grindwork") → make the two comprehension findings durable Rust probes, fanned out by model tier. +- **Fleet (edit-only, shared checkout, no worktree; orchestrator compiled centrally):** Sonnet agent → `crates/deepnsm/examples/freq_is_cosine.rs` (grindwork: the ρ test). Opus agent → `crates/deepnsm/examples/homograph_collapse.rs` (filigree: the 2³ homograph collapse). +- **Verified centrally** (scoped `cargo clippy/run --example ` so the operator's in-flight `gridlake_*` WIP was NOT touched): freq_is_cosine clippy-clean, **ρ=0.762** (reproduces the Python probe exactly; KILL floor 0.5); homograph_collapse clippy-clean, thought/left/means → distinct centroids (r53/r692, r152/r4559, r130/r1392), **809/809** verb∩noun homographs collapse. +- **Board:** `E-FREQ-IS-COSINE-REPLACEMENT-1` + `E-SURFACE-FORM-COLLAPSE-1` prepended. Commits `ea18a444` + `fb20a78e` on `claude/happy-hamilton-0azlw4` (PR #757, draft), on top of the `48cef03f` arm-discovery bridge (`E-ARM-DISCOVERY-REASONING-BRIDGE-1`). +- **Outcome:** the cosine-free ingestion-leg arc = 3 durable probes (discover premises from data → frequency-is-cosine distance → surface-form homograph collapse → reason), all measured + KILL-gated, no core change. Fleet pattern honored (Sonnet grind / Opus filigree). Left PR in draft — operator drives ready/merge. + ## 2026-07-19 — qualia-immersion / Chalmers-dissolution thesis + falsifiable exhibit — main thread - **Task:** operator ("break Chalmers hard problem by subjective experience of qualia through reasoning as immersion") → capture the endgame thesis + a falsifiable exhibit. diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index da731529..fc6b8068 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,43 @@ +## 2026-07-19 — E-FREQ-IS-COSINE-REPLACEMENT-1 — the per-subgenre word FREQUENCY DISTRIBUTION reproduces semantic-distance ordering with NO embedding and NO cosine-on-vectors (Spearman ρ=0.762 on the 8 in-repo genres alone) — "the centroid IS the cosine", the empirical ground under the palette256 / Morton / perturbation-shader distance + +**Status:** FINDING (built + measured in both Python and Rust, KILL-gated ρ≥0.5). **Confidence:** High. Deliverable: `crates/deepnsm/examples/freq_is_cosine.rs` (std-only, zero-dep). Operator thread: *"the frequency translates to cosine replacement centroid"* + *"there's no way this isn't relevant for distribution"* — offered as a testable claim, tested. + +**The claim, tested (not asserted).** A word's per-subgenre COCA frequency vector, `ln(1+PM)` per genre, z-scored across the vocabulary, Fisher-z'd, gives a pairwise distance that reproduces the DeepNSM README's NSM semantic-distance ordering at **Spearman ρ = 0.762** — with NO embedding and NO cosine computed on any learned vector. The distance is a property of WHERE the word's frequency lands, not of a comparison. Ground truth = the 8 README NSM pairs (`i/you` closest … `say/word` farthest); the extremes land correctly, the misranks (`before/after`, `the/this`) are exactly where 8 coarse genres lose the resolution the 96-subgenre palette recovers. + +**Why this is the ground under the distance substrate.** It is the empirical reason the operator's architecture is "relevant for distribution": because the frequency centroid IS the cosine, (a) there is nothing to compute pairwise — distance immaterializes (the Fisher-z is baked into palette-256 level placement, not stored k×k; cf. `bgz-tensor::fisher_z::FisherZTable` as the *materialized* reference the palette is calibrated against, and `contract::distance::similarity_z` / `mean_similarity_fisher` as the transform); and (b) it amortizes over the whole gridlake through the Morton 2bit×2bit 4×4 cascade rather than O(n²) pairwise, because every word is *placed*, not *compared*. The `6× palette256:palette256` perturbation-shader tenant (3 SPO + 3 AriGraph SPO), the ARM codebook oracle (`ρ=0.9973 vs cosine`), and CAM-PQ distance are all this one immaterialized-Fisher-z object. + +**Honest boundary (truth-architect).** This uses only the **8 coarse genres** committed to the repo (`lemmas_5k.csv`); the full **96-subgenre** palette (gitignored) is higher-resolution, so ρ=0.762 is a **FLOOR**, not a ceiling — the README's own NSM distances came from that 96-D CAM-PQ. The claim proven is the ORDERING (rank correlation), not absolute-distance identity (the scales differ). Additive: example only, no core change. **Cross-ref:** E-SURFACE-FORM-COLLAPSE-1 (the surface→lemma-centroid front door), E-ARM-DISCOVERY-REASONING-BRIDGE-1 (the ingestion leg), `contract::distance` (`similarity_z`), `bgz-tensor::fisher_z::FisherZTable`, the operator's palette256-immaterialized-Fisher-z thread. + +## 2026-07-19 — E-SURFACE-FORM-COLLAPSE-1 — the SPO 2³ role mask is a HOMOGRAPH-COLLAPSE OPERATOR: a surface word form is a superposition of (lemma, PoS) readings, and fixing the SPO slot collapses it to a UNIQUE lemma → a UNIQUE lemRank (the frequency-centroid address) — same surface, different centroid by role, measured on real COCA data + +**Status:** FINDING (built + measured, KILL-gated). **Confidence:** High for the collapse mechanism; the S/O→nominal, P→verbal role→PoS map is an explicit simplification (see boundary). Deliverable: `crates/deepnsm/examples/homograph_collapse.rs` (std-only, zero-dep). Operator thread: *"you can hold the meaning of all words in parallel and SPO 2^3 was always of superposition S_O / SP_ / _PO"* + *"please read the meaning of the word form abbreviation to understand how it could be used."* + +**The claim, made runnable.** A surface token is not a symbol — it is a SUPERPOSITION of its `(lemma, PoS)` senses (`thought` = {think·v, thought·n}). The SPO 2³ role mask is the COLLAPSE OPERATOR: the grammatical slot the token fills fixes a PoS class (Subject/Object → nominal `n`, Predicate → verbal `v`), which selects exactly ONE sense → ONE lemma → ONE `lemRank`. Since `lemRank` IS the frequency-centroid address (the same 5,050-lemma space the ρ=0.762 cosine-replacement lands in), **the same surface string addresses a DIFFERENT centroid depending on its role** — homograph disambiguation as superposition collapse, no parser weights. + +**Measured (`word_forms.csv`, 10,239 distinct surface forms):** +- 1,166 surfaces (11.39%) are ambiguous (>1 `(lemma,PoS)` sense); **1,164 are CROSS-PoS** (senses span ≥2 PoS letters) → collapsible by a role mask; only **2** are same-PoS residue (`found` = find·v / found·v) — the honest case a role mask ALONE cannot split. +- **809 verb∩noun homographs, ALL 809 (100%)** collapse to DISTINCT verbal-vs-nominal centroids under P vs S/O. +- Demonstrated (same surface, different centroid): `thought` → P **r53** (think·v) vs S/O **r692** (thought·n); `left` → P **r152** (leave·v) vs S/O **r4559** (left·n) — the `left·j` adjective sense is selected by NEITHER S/P/O mask (honest: adjective is a modifier role, not S/P/O); `means` → P **r130** vs S/O **r1392**. All three KILL-gated (each side unique, the two differ). + +**Why it matters for the ingestion leg.** This is the FRONT DOOR of the cosine-free ingestion (`E-ARM-DISCOVERY-REASONING-BRIDGE-1`): running text arrives inflected and ambiguous; `word_forms.csv` (11,460 forms → 5,050 lemma-centroids) resolves any surface form to its lemma-centroid, and the SPO role collapses the homograph superposition to the ONE centroid the slot selects. Surface text → role-collapsed centroid → palette/frequency distance, no embedding. + +**Honest boundary (truth-architect).** The S/O→nominal, P→verbal map is a SIMPLIFICATION — real syntactic slot assignment is the grammar/parser's job (a full FSM resolves which role a token fills). This probe demonstrates the COLLAPSE MECHANISM (a role mask disambiguates a homograph to a unique frequency-centroid), NOT full parsing. Additive: example only, no core change. **Cross-ref:** E-ARM-DISCOVERY-REASONING-BRIDGE-1 (the ingestion leg this feeds); the frequency-as-cosine-replacement finding (ρ=0.762); the 2³ SPO superposition (`S_O/SP_/_PO`); `deepnsm/word_frequency/word_forms.csv`. + +## 2026-07-19 — E-ARM-DISCOVERY-REASONING-BRIDGE-1 — the endgame's INGESTION leg wired: Aerial+ association-rule discovery mines premises from data (no LLM) → NARS-truth SPO → the self-directed loop derives the syllogism; and the memory-model comparison (RNN vs ring buffer vs Markov-NARS accretion) resolves to "the NARS revision arc IS the Markov trajectory" + +**Status:** FINDING (built + measured, 4 KILL-gates green) + FRAMING (memory-model comparison). **Confidence:** High for the bridge; the streaming-accretion half is plan-not-yet-wired (stated). Deliverable: `crates/lance-graph-arm-discovery/examples/discovery_to_reasoning.rs` (std-only, zero-dep, default features). Operator: *"did you wire Lance-graph-arm-discovery to arm discovery using aerial+ logic as functional routing of ideas … rnn vs ring buffer in Tesseract … does arm-discovery have any ice caking vs Markov chain context building and reasoning."* + +**The bridge (measured, no LLM, deterministic):** the two halves the workspace kept apart are now composed end-to-end — `extract_rules` (the REAL Aerial+ codebook probe: nearest consequent per feature, then support/confidence confirm on data) mines 4 `is_a` premises from 200 co-occurrence rows (`socrates→philosopher`, `plato→philosopher`, `philosopher→human`, `human→mortal`; each `f=1.00 c=0.98` earned from 50 co-occurrences) → `CandidateTriple::from_rule` routes them as NARS-truth SPO edges → the self-directed transitive-deduction loop (the shipped `self_directed_graph.rs` shape, std-only here) derives `socrates … mortal` (c=0.94) at a fixed point. **Nothing hand-asserted — the premises were MINED, the conclusion REASONED.** This is "read-without-an-LLM feeding reasoning-about-itself" as one float-free pipeline. + +**Two mining mechanics surfaced (correct the naive first cut):** (a) `extract_rules` picks the SINGLE codebook-nearest consequent per feature, so a multi-parent entity (`socrates` is_a philosopher AND greek) yields 0.5 rule-confidence and is DROPPED below the floor — a clean single-parent tree is required, and the reasoner supplies the chaining single-hop mining cannot; (b) the miner emits BOTH `f0→f1` and reversed `f1→f0` rules — the router must filter to the `is_a` (subject→object) direction. Both are in the example. + +**The memory-model comparison (the operator's RNN/ring-buffer/ice-cake question), grounded:** +- **RNN** (Tesseract LSTM, transcoded byte-parity in tesseract-rs `Lstm::forward` — cell `c`, output `h`, int8-requantized into the next timestep) = unbounded history compressed LOSSILY into a fixed-dim recurrent state, baked into weights: implicit, non-inspectable, non-revisable, non-replayable "ice cake in the weights." +- **Ring buffer** = explicit bounded FIFO, lossless-in-window, evicts the past. **arm-discovery literally contains one:** the `RatificationQueue` (bounded ring buffer, default 1024 candidates, backing onto Lance) in `streaming-arm-nars-discovery-v1.md:280`. +- **Markov stream + NARS revision** (temporal.rs sorted stream + the self-directed loop) = explicit, unbounded, revisable, REPLAYABLE (version-range reads). **arm-discovery's cross-window ice-caking IS this:** the streaming plan accretes each rule's truth by NARS-revising against its prior each window, and `I-SUBSTRATE-MARKOV` states *"the NARS revision arc IS the Markov trajectory"* (Chapman-Kolmogorov-consistent by construction). So arm-discovery's ice-caking and Markov context-building are the SAME mechanism. The workspace's bet: replace the RNN's opaque learned recurrence with explicit NARS-revised Markov accretion — same context-building function, inspectable/deterministic/no-weights. That is "AGI without LLM" at the memory layer. + +**Honest boundary + next step:** the shipped example is SINGLE-WINDOW stateless (the streaming cross-window NARS-revision accretion + the ring-buffer ratification queue are the plan `streaming-arm-nars-discovery-v1`, not yet wired); the reasoner is a std-only mirror of the real `TripletGraph` loop (kept zero-dep to honor the crate's isolation). **Named follow-up (grounds "read without LLM" in real text):** replace the synthetic co-occurrence rows with DeepNSM's real parse of philosophy text — DeepNSM's 4096-rank COCA vocab (98.4% coverage; academic register carries socrates/philosopher/mortal) parses text→SPO via its 6-state PoS FSM (no LLM), whose co-occurrences ARM then mines. **Cross-ref:** E-SELF-DIRECTED-GRAPH-1 (the reasoning leg), I-SUBSTRATE-MARKOV (NARS revision = Markov), `streaming-arm-nars-discovery-v1.md` (the streaming accretion plan), tesseract-rs `E-OCR-LSTM-1` (the RNN transcode). + ## 2026-07-19 — E-PERTURBATION-CONVERGENCE-1 — the five x265/H.268 arc findings CONVERGE onto ONE pinned equation (§4:108), real not rhyme; ratified by a 5-agent council; two invariants keep it from collapsing + one unproven "vertical" leg **Status:** FINDING (convergence council: 3 Sonnet harvesters [comma/phase, centroid-gridlake/256, axis-distinctness] + 2 Opus adjudicators [`convergence-architect`, `dilution-collapse-sentinel`]; both adjudicators INDEPENDENTLY ratified the same verdict — Opus for filigrane, Sonnet for grindwork per the standing model policy). Answers the operator's convergence question: *"Morton-cascade inverse-pyramid 4×4 ergonomics + centroid gridlake + vertical Pythagorean comma → perturbation-shader-cascade replayability WITHOUT materialization and WITHOUT collapsing."* diff --git a/crates/deepnsm/examples/freq_is_cosine.rs b/crates/deepnsm/examples/freq_is_cosine.rs new file mode 100644 index 00000000..da63fbe4 --- /dev/null +++ b/crates/deepnsm/examples/freq_is_cosine.rs @@ -0,0 +1,244 @@ +//! Claim under test: the per-subgenre word FREQUENCY DISTRIBUTION alone +//! reproduces NSM semantic-distance ordering, without training or querying +//! any embedding model -- i.e. "the frequency distribution is a cosine +//! replacement." +//! +//! Every COCA lemma in `word_frequency/lemmas_5k.csv` already carries an +//! 8-genre frequency profile (blog, web, TV/movie, spoken, fiction, +//! magazine, news, academic -- see `word_frequency/README.md`). This probe +//! treats that profile as if it were an embedding: log-compress it, +//! z-score every dimension across the whole vocabulary, take the cosine +//! between two words' profiles, and Fisher-z it into a distance. No neural +//! network, no learned weights -- only counts read straight out of a plain +//! frequency-count CSV. +//! +//! Reference: this was verified in Python at Spearman rho = 0.762 against +//! the 8 NSM pairs in `word_frequency/README.md` sect. "Semantic Distance +//! Verification". This example reproduces that measurement in pure-std +//! Rust and is KILL-gated at rho < 0.5. +//! +//! ## Algorithm +//! +//! 1. Parse `lemmas_5k.csv`; for each distinct lowercased lemma (first +//! occurrence wins -- the file is rank-ascending, so that is also the +//! highest-frequency part of speech), take the 8 per-million genre +//! columns (`blogPM..acadPM`, the last 8 of 25) and log-compress each: +//! `v[d] = ln(1 + PM[d])`. +//! 2. Z-score every dimension across the WHOLE vocabulary (population +//! mean/stddev; a ~zero-stddev dimension is left unscaled). +//! 3. Distance between two words = Fisher-z of their z-scored cosine: +//! `c = dot(za, zb) / (|za| * |zb|)`, `dist = -0.5 * ln((1+c)/(1-c))`, +//! with `c` clamped to `[-0.9999, 0.9999]`. More negative = closer. +//! 4. Rank the 8 NSM reference pairs' README distances and this run's +//! Fisher-z distances; the Spearman rho between the two rankings is +//! the scored claim. +//! +//! ## Run +//! +//! ```bash +//! cargo run --manifest-path crates/deepnsm/Cargo.toml --example freq_is_cosine +//! ``` + +use std::collections::{HashMap, HashSet}; + +/// The 8 per-million genre columns of `lemmas_5k.csv`, in file order: +/// blog, web, TV/movie, spoken, fiction, magazine, news, academic. +const DIMS: usize = 8; + +/// 0-based index of the first `*PM` genre column. The header is +/// `rank,lemma,PoS,freq,perMil,%caps,%allC,range,disp` (9 fields) + +/// 8 raw genre counts + 8 `*PM` genre columns = 25 fields total. +const PM_START: usize = 17; + +/// One vocabulary entry: the lemma text plus its 8-genre frequency +/// profile. Holds raw `ln(1+PM)` values on load; [`zscore_in_place`] +/// overwrites `dims` with the z-scored form. +struct WordVec { + lemma: String, + dims: [f64; DIMS], +} + +/// Parse `lemmas_5k.csv` into one [`WordVec`] per distinct lowercased +/// lemma, keeping the FIRST row seen for each (the file is rank-ascending, +/// so "first occurrence" is also "highest-frequency part of speech"). +/// Missing or unparseable genre fields fall back to `0.0`. +fn load_vectors(csv_path: &str) -> Vec { + let text = std::fs::read_to_string(csv_path) + .unwrap_or_else(|e| panic!("failed to read {csv_path}: {e}")); + + let mut seen: HashSet = HashSet::new(); + let mut out = Vec::new(); + for line in text.lines().skip(1) { + let fields: Vec<&str> = line.split(',').collect(); + if fields.len() < PM_START + DIMS { + continue; + } + let lemma = fields[1].trim().to_lowercase(); + if lemma.is_empty() || !seen.insert(lemma.clone()) { + continue; + } + let dims: [f64; DIMS] = std::array::from_fn(|d| { + let raw: f64 = fields[PM_START + d].trim().parse().unwrap_or(0.0); + (1.0 + raw).ln() + }); + out.push(WordVec { lemma, dims }); + } + out +} + +/// Z-score every dimension across the whole vocabulary, in place +/// (population mean/stddev; a ~zero-stddev dimension is left unscaled so +/// the division never blows up). +fn zscore_in_place(vectors: &mut [WordVec]) { + let n = vectors.len() as f64; + + let mean: [f64; DIMS] = + std::array::from_fn(|d| vectors.iter().map(|v| v.dims[d]).sum::() / n); + + let std_dev: [f64; DIMS] = std::array::from_fn(|d| { + let var = vectors + .iter() + .map(|v| (v.dims[d] - mean[d]).powi(2)) + .sum::() + / n; + let s = var.sqrt(); + if s.abs() < 1e-12 { + 1.0 + } else { + s + } + }); + + for v in vectors.iter_mut() { + v.dims = std::array::from_fn(|d| (v.dims[d] - mean[d]) / std_dev[d]); + } +} + +/// Cosine similarity between two z-scored 8-dim profiles; `0.0` if either +/// norm is ~zero (guards a degenerate all-zero profile). +fn cosine(a: &[f64; DIMS], b: &[f64; DIMS]) -> f64 { + let dot: f64 = a.iter().zip(b.iter()).map(|(x, y)| x * y).sum(); + let norm_a = a.iter().map(|x| x * x).sum::().sqrt(); + let norm_b = b.iter().map(|x| x * x).sum::().sqrt(); + let denom = norm_a * norm_b; + if denom < 1e-9 { + 0.0 + } else { + dot / denom + } +} + +/// Fisher-z transform of a cosine similarity into a distance: more +/// negative is closer, larger is farther. `c` is clamped away from `+-1` +/// so the `ln` stays finite. +fn fisher_z_distance(c: f64) -> f64 { + let c = c.clamp(-0.9999, 0.9999); + -0.5 * ((1.0 + c) / (1.0 - c)).ln() +} + +/// 1-based ranks of `values`, ascending; tied values (within `1e-12`) +/// share the average rank of their tie group. +fn ranks(values: &[f64]) -> Vec { + let n = values.len(); + let mut idx: Vec = (0..n).collect(); + idx.sort_by(|&i, &j| values[i].total_cmp(&values[j])); + + let mut out = vec![0.0f64; n]; + let mut i = 0; + while i < n { + let mut j = i; + while j + 1 < n && (values[idx[j + 1]] - values[idx[i]]).abs() < 1e-12 { + j += 1; + } + let avg_rank = (i + 1 + j + 1) as f64 / 2.0; + for &k in &idx[i..=j] { + out[k] = avg_rank; + } + i = j + 1; + } + out +} + +/// Spearman rank correlation between two equal-length value slices. +fn spearman_rho(xs: &[f64], ys: &[f64]) -> f64 { + let n = xs.len() as f64; + let rx = ranks(xs); + let ry = ranks(ys); + let sum_d2: f64 = rx.iter().zip(ry.iter()).map(|(a, b)| (a - b).powi(2)).sum(); + 1.0 - (6.0 * sum_d2) / (n * (n * n - 1.0)) +} + +fn main() { + let csv_path = concat!(env!("CARGO_MANIFEST_DIR"), "/word_frequency/lemmas_5k.csv"); + let mut vectors = load_vectors(csv_path); + let n_words = vectors.len(); + + println!("── FREQUENCY DISTRIBUTION AS A COSINE REPLACEMENT ────────────────"); + println!(" loaded {n_words} distinct lemmas from {csv_path}"); + + zscore_in_place(&mut vectors); + + let mut index: HashMap<&str, usize> = HashMap::with_capacity(vectors.len()); + for (i, v) in vectors.iter().enumerate() { + index.insert(v.lemma.as_str(), i); + } + + // The 8 NSM reference pairs, ascending README semantic distance + // (word_frequency/README.md sect. "Semantic Distance Verification"). + let pairs: [(&str, &str, f64); 8] = [ + ("i", "you", 3.32), + ("think", "know", 3.85), + ("before", "after", 4.46), + ("live", "die", 5.35), + ("good", "bad", 6.60), + ("think", "big", 8.16), + ("the", "this", 11.47), + ("say", "word", 15.22), + ]; + + let mut freq_dist = Vec::with_capacity(pairs.len()); + let mut readme_dist = Vec::with_capacity(pairs.len()); + + println!(); + println!( + " {:<14} {:>14} {:>14}", + "pair", "freq_dist", "readme_dist" + ); + for &(a, b, readme) in &pairs { + let ia = *index + .get(a) + .unwrap_or_else(|| panic!("'{a}' not found in vocabulary")); + let ib = *index + .get(b) + .unwrap_or_else(|| panic!("'{b}' not found in vocabulary")); + let dist = fisher_z_distance(cosine(&vectors[ia].dims, &vectors[ib].dims)); + let label = format!("{a}/{b}"); + println!(" {label:<14} {dist:>14.4} {readme:>14.2}"); + freq_dist.push(dist); + readme_dist.push(readme); + } + + let rho = spearman_rho(&freq_dist, &readme_dist); + println!(); + println!("Spearman rho (freq-distance vs README semantic ordering): {rho:.3}"); + + if rho < 0.5 { + eprintln!( + " ✗ KILL: frequency distribution does NOT reproduce semantic ordering (rho={rho:.3})" + ); + std::process::exit(1); + } + + println!(); + println!(" ✓ claim holds: rho={rho:.3} >= 0.5 (this probe's KILL floor)."); + println!(" the per-subgenre word FREQUENCY DISTRIBUTION reproduces the NSM"); + println!(" semantic-distance ordering. No embeddings, no cosine-on-vectors --"); + println!(" the distance above is a property of the frequency distribution"); + println!(" alone (ln(1+PM) per genre, z-scored across the vocabulary, Fisher-z'd)."); + println!(); + println!(" honest boundary: this uses only the 8 coarse GENRES committed to"); + println!(" this repo (blog/web/TVM/spoken/fiction/magazine/news/academic)."); + println!(" the full 96-SUBGENRE palette (word_frequency/README.md, gitignored)"); + println!(" is higher-resolution, so this rho is a FLOOR on what the frequency"); + println!(" distribution can do here, not a ceiling."); +} diff --git a/crates/deepnsm/examples/homograph_collapse.rs b/crates/deepnsm/examples/homograph_collapse.rs new file mode 100644 index 00000000..be006ae1 --- /dev/null +++ b/crates/deepnsm/examples/homograph_collapse.rs @@ -0,0 +1,259 @@ +//! Homograph collapse — the SPO 2³ role mask as a superposition collapse operator. +//! +//! CLAIM. A surface word form is a SUPERPOSITION of readings: it maps to a SET +//! of `(lemma, PoS)` senses. The SPO 2³ role mask (Subject / Predicate / Object) +//! is the COLLAPSE OPERATOR — fixing which grammatical role a token fills selects +//! one reading, collapsing the superposition to a UNIQUE lemma, hence a UNIQUE +//! `lemRank` (the frequency-centroid address in the COCA-ranked vocabulary). So +//! the SAME surface string lands on DIFFERENT centroids depending on its slot. +//! This grounds "hold the meaning of all words in parallel, collapse by role" +//! on real data. +//! +//! Minimal collapse rule (a SIMPLIFICATION — see HONEST BOUNDARY below): +//! S → expects a nominal ('n'); P → expects a verbal ('v'); O → nominal ('n'). +//! Given `(surface, role)`, keep the senses whose PoS matches the role's class; +//! the collapse is UNIQUE iff exactly one sense survives. +//! +//! Data: `word_frequency/word_forms.csv` (COCA), header +//! `lemRank,lemma,PoS,lemFreq,wordFreq,word`. Std-only, hand-parsed. +//! +//! Run: cargo run --manifest-path crates/deepnsm/Cargo.toml --example homograph_collapse + +use std::collections::HashMap; + +/// One reading of a surface word: a `(lemma, PoS)` sense addressed by its +/// lemma-frequency rank. `lem_rank` IS the centroid address — the thing a role +/// collapse selects out of the superposition. +#[derive(PartialEq, Eq)] +struct Sense { + lemma: String, + pos: char, + lem_rank: u32, +} + +/// A grammatical slot in the SPO 2³ mask. Each slot fixes the PoS *class* the +/// token must fill: S and O are nominal, P is verbal. Fixing the role is the +/// COLLAPSE OPERATOR that selects one reading of a homograph. +#[derive(Clone, Copy)] +enum Role { + Subject, + Predicate, + Object, +} + +impl Role { + /// The PoS letter this role's class expects: 'v' (verbal) for P, else 'n'. + fn expected_pos(self) -> char { + match self { + Role::Predicate => 'v', + Role::Subject | Role::Object => 'n', + } + } + + fn label(self) -> &'static str { + match self { + Role::Subject => "S", + Role::Predicate => "P", + Role::Object => "O", + } + } +} + +/// Result of applying a role mask to a surface word's sense superposition. +#[derive(Clone, Copy)] +enum Collapse<'a> { + /// Superposition collapsed to a single reading: `(lemma, lemRank centroid)`. + Unique(&'a str, u32), + /// More than one sense of the expected class — the role mask ALONE is + /// insufficient (the same-PoS residue, e.g. `found` = find.v / found.v). + Ambiguous(usize), + /// No sense of the expected class fills this role. + None, +} + +/// Apply a role mask: keep only senses whose PoS matches the role's class, then +/// report whether the superposition collapsed to a UNIQUE frequency-centroid. +fn collapse(senses: &[Sense], role: Role) -> Collapse<'_> { + let want = role.expected_pos(); + let mut hits = senses.iter().filter(|s| s.pos == want); + match (hits.next(), hits.next()) { + (Some(s), None) => Collapse::Unique(s.lemma.as_str(), s.lem_rank), + (None, _) => Collapse::None, + (Some(_), Some(_)) => Collapse::Ambiguous(senses.iter().filter(|s| s.pos == want).count()), + } +} + +/// Parse `word_forms.csv` into `lowercased surface → Vec` (identical +/// tuples deduplicated). Hand-split on ',' — deepnsm is std-only, zero-dep. +fn parse_forms(csv: &str) -> HashMap> { + let mut map: HashMap> = HashMap::new(); + for line in csv.lines().skip(1) { + // lemRank,lemma,PoS,lemFreq,wordFreq,word + let f: Vec<&str> = line.split(',').collect(); + if f.len() < 6 { + continue; + } + let Ok(lem_rank) = f[0].parse::() else { + continue; + }; + let Some(pos) = f[2].chars().next() else { + continue; + }; + let sense = Sense { + lemma: f[1].to_string(), + pos, + lem_rank, + }; + let bucket = map.entry(f[5].to_ascii_lowercase()).or_default(); + if !bucket.contains(&sense) { + bucket.push(sense); + } + } + map +} + +/// One line of a demonstration: how a role mask collapses the superposition. +fn describe(senses: &[Sense], role: Role) -> String { + let lab = role.label(); + let pos = role.expected_pos(); + let class = if pos == 'v' { "verbal" } else { "nominal" }; + match collapse(senses, role) { + Collapse::Unique(lemma, rank) => { + format!("{lab} → {class:<7} → {lemma}.{pos} → centroid r{rank}") + } + Collapse::Ambiguous(n) => { + format!( + "{lab} → {class:<7} → AMBIGUOUS: {n} senses of class '{pos}' (mask alone fails)" + ) + } + Collapse::None => format!("{lab} → {class:<7} → (no {class} reading present)"), + } +} + +/// Demonstrate + KILL-GATE one cross-PoS homograph. The P-slot must collapse to +/// a unique verbal centroid, the S/O-slot to a unique nominal centroid, and the +/// two addresses must DIFFER — else the superposition did not split under role +/// masking and the probe is falsified. +fn demonstrate(map: &HashMap>, surface: &str) { + let Some(senses) = map.get(surface) else { + println!( + " ✗ KILL: {surface} did not collapse to two distinct unique centroids under role masking" + ); + std::process::exit(1); + }; + + let bundle: Vec = senses + .iter() + .map(|s| format!("({}, {}, r{})", s.lemma, s.pos, s.lem_rank)) + .collect(); + println!(" '{surface}' superposition = {{ {} }}", bundle.join(", ")); + for role in [Role::Subject, Role::Predicate, Role::Object] { + println!(" {}", describe(senses, role)); + } + + // KILL GATE: P-collapse and S/O-collapse each UNIQUE and DIFFERENT. + let p = collapse(senses, Role::Predicate); + let so = collapse(senses, Role::Subject); // O is identical (also nominal) + if let (Collapse::Unique(_, pr), Collapse::Unique(_, sr)) = (p, so) { + if pr != sr { + println!(" ✓ same surface, DISTINCT centroids by role: P r{pr} ≠ S/O r{sr}\n"); + return; + } + } + println!( + " ✗ KILL: {surface} did not collapse to two distinct unique centroids under role masking" + ); + std::process::exit(1); +} + +fn main() { + let csv = std::fs::read_to_string(concat!( + env!("CARGO_MANIFEST_DIR"), + "/word_frequency/word_forms.csv" + )) + .expect("read word_forms.csv"); + let map = parse_forms(&csv); + + println!( + "── HOMOGRAPH COLLAPSE · the SPO 2³ role mask as a superposition collapse operator ──" + ); + println!( + " A surface word is a SUPERPOSITION of (lemma, PoS) readings. Fixing the SPO slot it" + ); + println!( + " fills is the COLLAPSE OPERATOR: role → PoS class → one lemma → one lemRank centroid." + ); + println!( + " mask: {} & {} → nominal | {} → verbal\n", + Role::Subject.label(), + Role::Object.label(), + Role::Predicate.label() + ); + + // ── census: how much of the vocabulary is a collapsible superposition? ── + let distinct = map.len(); + let mut ambiguous = 0usize; + let mut cross_pos = 0usize; + let mut verb_noun = 0usize; + let mut collapsible = 0usize; + for senses in map.values() { + if senses.len() > 1 { + ambiguous += 1; + let mut letters: Vec = senses.iter().map(|s| s.pos).collect(); + letters.sort_unstable(); + letters.dedup(); + if letters.len() >= 2 { + cross_pos += 1; + } + } + let has_v = senses.iter().any(|s| s.pos == 'v'); + let has_n = senses.iter().any(|s| s.pos == 'n'); + if has_v && has_n { + verb_noun += 1; + if let (Collapse::Unique(_, pr), Collapse::Unique(_, sr)) = ( + collapse(senses, Role::Predicate), + collapse(senses, Role::Subject), + ) { + if pr != sr { + collapsible += 1; + } + } + } + } + let same_pos = ambiguous - cross_pos; + let pct = 100.0 * ambiguous as f64 / distinct as f64; + let collaps_pct = 100.0 * collapsible as f64 / verb_noun as f64; + + println!("── AMBIGUITY CENSUS · word_frequency/word_forms.csv ──"); + println!(" distinct surface forms : {distinct}"); + println!( + " ambiguous (>1 (lemma,PoS) sense) : {ambiguous} ({pct:.2}% of surfaces)" + ); + println!(" ├─ CROSS-PoS (senses span ≥2 PoS letters) : {cross_pos} ← collapsible by a role mask"); + println!(" └─ same-PoS residue (e.g. found=find.v/found.v): {same_pos} ← role mask ALONE cannot split"); + println!(" verb∩noun homographs (≥1 verb AND ≥1 noun) : {verb_noun}"); + println!(" └─ collapse to distinct v/n centroids under P-vs-S/O : {collapsible} ({collaps_pct:.0}%)\n"); + + // ── the three cross-PoS demonstrations (also the kill gate) ── + println!("── DEMONSTRATION · same surface, different SPO slot → different centroid ──"); + for surface in ["thought", "left", "means"] { + demonstrate(&map, surface); + } + + // ── synthesis + honest boundary ── + println!("── SYNTHESIS ──"); + println!(" Each surface word is held as the SUPERPOSITION of its readings. The SPO 2³ role"); + println!(" mask is the collapse operator: the slot a token fills fixes one PoS class → one"); + println!(" lemma → one lemRank, a unique frequency-centroid. 'thought' addresses r53 as a"); + println!(" Predicate but r692 as a Subject/Object — same string, different centroid, chosen"); + println!(" by role alone; likewise left (r152/r4559) and means (r130/r1392). Across COCA,"); + println!(" {collapsible} verb∩noun homographs collapse this way.\n"); + println!(" HONEST BOUNDARY: the S/O→nominal, P→verbal mapping is a SIMPLIFICATION. Real"); + println!(" syntactic role assignment is the parser's job — a full grammar decides which slot"); + println!(" a token actually fills, and resolves the same-PoS residue (found=find.v/found.v)"); + println!(" this mask cannot split. This probe demonstrates the COLLAPSE MECHANISM — a role"); + println!(" mask disambiguating a homograph to a unique centroid — not full parsing. It shows"); + println!( + " the 2³ SPO mask acting as a homograph-collapse operator on real surface-form data." + ); +} diff --git a/crates/lance-graph-arm-discovery/examples/discovery_to_reasoning.rs b/crates/lance-graph-arm-discovery/examples/discovery_to_reasoning.rs new file mode 100644 index 00000000..2bde605e --- /dev/null +++ b/crates/lance-graph-arm-discovery/examples/discovery_to_reasoning.rs @@ -0,0 +1,263 @@ +//! `discovery_to_reasoning` — the **ingestion leg of the endgame**: read a graph's +//! premises OUT OF DATA with no LLM (Aerial+ association-rule discovery), route +//! them into the graph as NARS-truth SPO edges, then let the graph reason about +//! itself to a conclusion the data only IMPLIED. +//! +//! This wires the two halves the workspace had kept apart: +//! - **discovery** (this crate) — mines `(X → Y)` rules from tabular co-occurrence +//! via the integer codebook probe (float-free, deterministic), with NARS truth +//! `(frequency, confidence)` earned from the evidence counts; +//! - **self-directed reasoning** (the loop below, std-only) — the same shape as +//! lance-graph's shipped `examples/self_directed_graph.rs` (`TripletGraph` + +//! `infer_deductions`): transitive NARS deduction to a fixed point. Kept to +//! ~30 std-only lines here so this crate stays zero-dep; the deduction is +//! identical (`f = f₁f₂, c = f₁f₂c₁c₂`). +//! +//! ```text +//! tabular rows (co-occurrence) +//! ─► extract_rules (codebook probe picks the nearest consequent per feature, +//! then confirms support/confidence on the data) [REAL discovery] +//! ─► CandidateTriple::from_rule → (s, is_a, o, f, c) NARS-truth SPO +//! ─► the graph reasons about itself: transitive deduction to a fixed point +//! ─► CONCLUSION derived from DISCOVERED premises: `socrates … mortal` +//! ``` +//! +//! Nothing is hand-asserted: every `is_a` edge the reasoner chains was MINED from +//! the data by Aerial+ logic, its confidence the discovery's own evidence count. +//! (The multi-path witness — a belief corroborated by an independent path — needs +//! a fork, which splits a subject's rule-confidence below the mining floor; it is +//! demonstrated in the shipped `self_directed_graph.rs`, not re-proven here.) +//! +//! ```sh +//! cargo run --manifest-path crates/lance-graph-arm-discovery/Cargo.toml \ +//! --example discovery_to_reasoning +//! ``` + +use lance_graph_arm_discovery::{ + extract_rules, CandidateTriple, Dataset, ExtractParams, FeatureSpec, FeedProjector, Item, + TopKDistance, NARS_PERSONALITY_K, +}; + +/// A shared entity vocabulary: the SAME string can be a subject (feature 0) in one +/// row and an object (feature 1) in another, so the mined `is_a` edges chain. +const VOCAB: [&str; 5] = ["socrates", "plato", "philosopher", "human", "mortal"]; + +/// Names a mined rule's items back to entity strings → `(subject, is_a, object)`. +/// This is the "functional routing of ideas": a discovered `(feat0=cat_i → feat1=cat_j)` +/// association becomes the graph edge `VOCAB[i] --is_a--> VOCAB[j]`. +struct EntityProjector; +impl FeedProjector for EntityProjector { + fn subject(&self, antecedent: &[Item]) -> String { + VOCAB[antecedent[0].category as usize].to_string() + } + fn predicate(&self) -> String { + "is_a".to_string() + } + fn object(&self, consequent: &[Item]) -> String { + VOCAB[consequent[0].category as usize].to_string() + } +} + +/// A NARS-truth `is_a` fact the graph holds (owned microcopy). +#[derive(Clone)] +struct Fact { + s: String, + o: String, + f: f32, + c: f32, + path: String, // "" = mined base fact; else the intermediate it was derived through +} + +fn concluded(facts: &[Fact], s: &str, o: &str) -> bool { + facts.iter().any(|x| x.s == s && x.o == o) +} + +fn main() { + // ── 1. DATA — co-occurrence rows (each subject has ONE parent ⇒ clean 100% + // rule confidence; the reasoner supplies the chaining the single-hop mining + // cannot). Each pair ×50 = the evidence the mining counts. ── + let spec = FeatureSpec::new(vec![VOCAB.len() as u32, VOCAB.len() as u32]); + let pairs = [ + (0u32, 2u32), // socrates is_a philosopher + (1, 2), // plato is_a philosopher + (2, 3), // philosopher is_a human + (3, 4), // human is_a mortal + ]; + let mut rows = Vec::new(); + for &(s, o) in &pairs { + for _ in 0..50 { + rows.push(vec![s, o]); + } + } + let data = Dataset::new(spec.clone(), rows); + + // ── 2. DISCOVER — the REAL Aerial+ codebook probe (float-free, deterministic). + // The splat oracle places each subject next to the object it co-occurs with; + // extract_rules confirms support/confidence on the data. ── + let edges: Vec<(Item, Item, u32)> = pairs + .iter() + .map(|&(s, o)| (Item::new(0, s), Item::new(1, o), 1)) + .collect(); + let oracle = TopKDistance::new(spec, u32::MAX, &edges); + let rules = extract_rules( + &oracle, + &data, + &ExtractParams { + theta: 2, + max_antecedent: 1, + min_support_ppm: 100_000, // ≥10% of the window + min_confidence_ppm: 700_000, // ≥70% rule confidence + }, + ); + + println!("── discovery_to_reasoning : read the premises from DATA (no LLM), then reason ──\n"); + println!( + " Aerial+ mined association rules from {} co-occurrence rows (nothing hand-asserted):", + pairs.len() * 50 + ); + + // ── 3. ROUTE — mined subject→object rules → NARS-truth `is_a` edges (the seed). + // Keep only the s→o direction (feature 0 → feature 1); the miner also emits + // reversed o→s rules, which are not the `is_a` reading. ── + let proj = EntityProjector; + let mut facts: Vec = Vec::new(); + for r in &rules { + let forward = r.antecedent.len() == 1 + && r.consequent.len() == 1 + && r.antecedent[0].feature == 0 + && r.consequent[0].feature == 1; + if !forward { + continue; + } + let t: CandidateTriple = CandidateTriple::from_rule(r, &proj, NARS_PERSONALITY_K); + println!( + " {:>11} is_a {:<12} f={:.2} c={:.2} (mined from {} co-occurrences)", + t.s, t.o, t.f, t.c, r.cooccur + ); + facts.push(Fact { + s: t.s, + o: t.o, + f: t.f, + c: t.c, + path: String::new(), + }); + } + let mined = facts.len(); + + // ── 4. THE GRAPH REASONS ABOUT ITSELF over the DISCOVERED facts: transitive + // NARS deduction (A is_a B, B is_a C ⇒ A is_a C) to a fixed point. ── + let mut derived: Vec = Vec::new(); + let mut socrates_mortal_conf: Option = None; + let mut rested = false; + for _round in 0..16 { + let mut fresh: Option = None; + 'scan: for a in &facts { + for b in &facts { + if a.o != b.s || a.s == b.o { + continue; // need A→B, B→C, no self-loop + } + if !concluded(&facts, &a.s, &b.o) { + fresh = Some(Fact { + s: a.s.clone(), + o: b.o.clone(), + f: a.f * b.f, + c: a.f * b.f * a.c * b.c, + path: a.o.clone(), + }); + break 'scan; + } + } + } + match fresh { + Some(nf) => { + if nf.s == "socrates" && nf.o == "mortal" { + socrates_mortal_conf = Some(nf.c); + } + derived.push(format!( + "{} is_a {} (via {}) f={:.2} c={:.2}", + nf.s, nf.o, nf.path, nf.f, nf.c + )); + facts.push(nf); + } + None => { + rested = true; + break; + } + } + } + + println!("\n the graph then reasoned about itself over those mined facts:"); + for d in &derived { + println!(" + {d}"); + } + + // ── MEASUREMENTS (the bridge is load-bearing only if these hold) ── + let mined_ok = mined >= 4; + let grew = facts.len() > mined && concluded(&facts, "socrates", "human"); + let syllogism = concluded(&facts, "socrates", "mortal"); + + println!( + "\n mined ≥4 is_a premises from data (none hand-asserted): {}", + yn(mined_ok) + ); + println!( + " the graph grew by reasoning ({mined} mined → {} total beliefs): {}", + facts.len(), + yn(grew) + ); + println!( + " ★ SYLLOGISM from DISCOVERED premises — `socrates … mortal` (c={:.2}), no LLM: {}", + socrates_mortal_conf.unwrap_or(0.0), + yn(syllogism) + ); + println!(" reached a coherent fixed point (rested): {}", yn(rested)); + + let kills: Vec<&str> = [ + ( + mined_ok, + "fewer than 4 forward rules mined — discovery under-fired", + ), + ( + grew, + "reasoning did not grow the graph past the mined facts", + ), + ( + syllogism, + "failed to derive socrates→mortal from mined premises", + ), + (rested, "did not reach a fixed point"), + ] + .iter() + .filter_map(|&(ok, m)| (!ok).then_some(m)) + .collect(); + + println!("\n── the bridge, stated plainly ──"); + if kills.is_empty() { + println!( + " The premises were not written by a human or an LLM — Aerial+ MINED them from data" + ); + println!( + " as truth-valued rules, routed them into the graph as is_a edges, and the graph" + ); + println!( + " reasoned over its own mined tissue to a conclusion the data only implied. That is" + ); + println!( + " the ingestion leg of the endgame: reading-without-an-LLM feeding reasoning-about-" + ); + println!(" itself — discovery and deduction as one float-free, deterministic pipeline."); + } else { + for k in &kills { + println!(" ✗ KILL: {k}"); + } + std::process::exit(1); + } +} + +fn yn(b: bool) -> &'static str { + if b { + "YES ✓" + } else { + "NO ✗" + } +}