Can a small LLM fine-tuned as a generative retriever (query → note-id, à la DSI) beat a strong
embedding baseline on a small, densely-linked, franglais personal-memory corpus — especially on
associative ("two-hop") and multi-answer recall, where embedding retrieval is documented to be weak?
A self-contained, reproducible experiment that pits four retrieval arms against each other on the same held-out, evaluated with a protocol built to measure genuine generalization (no verbatim match, no memorization shortcut). A negative result is a valid result here — the numbers decide.
📄 Writeup & verdict → paper/PAPER.md · 🧪 Brief → EXPERIMENT.md
| Arm | Idea | Where it runs |
|---|---|---|
| A — Embedding baseline | multilingual-e5-small dense + BM25, cosine → top-k slugs. No training. The bar to beat. |
CPU |
| B — Generative router (DSI) | LoRA-fine-tune a small LM to emit the note slug from a query; the weights become the index. Inference ranks every valid slug by likelihood — 0 hallucinated ids by construction. | AMD RX 9070 XT via DirectML |
| C — Query decomposition | Split a multi-answer query into K single-target sub-queries, retrieve each via A or B, union → coverage. Composes with A or B. | local LLM + A/B |
| D — From-scratch char-CNN (control) | A tiny character-level CNN (~1–19M params, no pretraining) trained on the same query→slug pairs. Isolates whether the associative win is the routing formulation or the base model's pretrained semantics. | CPU (minutes) |
Evaluated on 1,031 LLM-generated, never-trained held-out queries (656 symptom-side + 375 independent second-hop), source note excluded for associative scoring. Same metric code for every arm:
| Subset | metric | Arm A — embedding | router 0.5B | router 1.5B | router 3B |
|---|---|---|---|---|---|
| symptom (cause↔symptom gap) | HIT@1 / HIT@10 | 0.401 / 0.669 | 0.178 / 0.593 | 0.364 / 0.794 | 0.456 / 0.883 |
| associative (independent 2-hop) | HIT@1 / HIT@10 | 0.275 / 0.579 | 0.245 / 0.640 | 0.499 / 0.877 | 0.515 / 0.893 |
| multi-answer | coverage@10 | 0.677 | 0.493 | 0.579 | 0.590 |
Invalid-slug rate 0 everywhere. The story is size-dependent and turns on the associative case: the 0.5B router loses overall (it only ties associative), but the 1.5B router decisively beats the embedding baseline on associative two-hop recall — paired bootstrap HIT@1 +0.224 (95% CI [+0.160, +0.291]) and HIT@10 +0.299 ([+0.245, +0.355]), both clear of zero — and wins symptom HIT@10, while still losing multi-answer coverage (0.579 vs 0.677). So H1 is partially supported (associative yes, multi-answer no), and the win holds at 3B (associative +0.315 vs baseline).
H2 — the 0.5B→1.5B→3B curve gives a clean dissociation. Both two-hop tasks climb steeply to 1.5B,
then split: the associative advantage flattens — 3B adds no significant gain (paired HIT@1
+0.016, 95% CI [−0.037, +0.075]; HIT@10 +0.016, [−0.019, +0.051]) — while symptom keeps climbing
(HIT@1 +0.092, [+0.050, +0.133]). Associative recall is a formulation win (learning the link graph,
which a mid-size model captures — a from-scratch char-CNN, Arm D, even wins it); symptom is a capacity
win that keeps paying for scale. See paper/router_scaling.svg.
Arm D nails it down. A from-scratch character-level CNN with zero pretraining also beats the
embedding baseline on associative recall (converged 1.3M-param model: HIT@10 0.643 vs 0.579) and
scales cleanly with size (paper/arm_d_scaling.svg) — so the associative
advantage really is the routing formulation (learning
the link graph), not pretrained semantics. Only symptom recall stays weak from scratch (it needs the
pretrained model). This points to a fully local, laptop-CPU deployment (paper/PAPER.md §12).
And it's the cheapest arm by far. Cost per query (FLOPs = hardware-independent; latency = same CPU): the char-CNN Pareto-dominates the baseline — ~25× fewer FLOPs and ~8× lower latency (2 ms/query) and better associative recall — while the 0.5–3B router pays a ~3,000–6,000× compute premium (it does one forward pass per note, O(N), so the multiplier grows with the corpus) for its top-end recall.
| arm (assoc HIT@10) | FLOPs/query | latency (CPU) |
|---|---|---|
| char-CNN 2M (0.643) | 0.24 GFLOP (25× less) | 1.96 ms (8× faster) |
| baseline e5+BM25 (0.579) | 6.6 GFLOP | 15.9 ms |
| router 1.5B (0.877) | 20 TFLOP (~3,000×) | 163 s |
| router 3B (0.893) | 40 TFLOP (~6,000×) | 319 s |
Complexity per query (N = notes, L = query length): baseline and char-CNN call the model O(1) times
(one encode / one conv forward) then score all notes in a cheap O(N) tail; the router calls the model
O(N) times — one full forward per note — so its cost is O(N·L) and grows with the corpus. The
char-CNN is also O(L) in query length (convolutions, no O(L²) attention). Details in paper/PAPER.md §5.5.
The comparison is deliberately asymmetric (the router is trained on the corpus link graph, the baseline is zero-shot on it) — so the win means a router that learned the associations generalizes to novel second-hop phrasings cosine can't bridge. No arm clears the absolute bar (HIT@1 > 0.80).
Query decomposition (Arm C) does not help — exhaustively tested. It lowered coverage@10 at every retriever (holistic 0.677 → C×A 0.598 → C×B 0.562); routing the sub-queries through the stronger router made it worse (a sharp single-best ranker unions poorly), and even RRF-fusing the decomposed signal on top of the baseline gave no significant gain (cov@10 +0.023, CI [−0.007,+0.054]). The cause is structural — a cardinality ceiling: 94% of queries have more gold members than sub-queries, so splitting the output slots dilutes coverage. A clean negative, now measured (C×B run) not argued.
Full per-k tables, paired CIs, the Arm D char-CNN, and the verdict are in paper/PAPER.md.
The protocol is built so that "B beats A" can only reflect real retrieval skill, not a measurement artifact:
- Verbatim text is train-only. Note titles/summaries anchor the model in training; they are never used as eval queries (otherwise the query is the document → a meaningless HIT@1 = 1.000).
- Held-out queries are generated, not split. A Sonnet workflow writes symptom-side paraphrases (different words than the note) and independent second-hop questions (answer = the linked note, without quoting the source) — zero verbatim text, never seen in training.
- Associative scoring excludes the source note A, so it measures reaching the linked B, not "ranking B above its own text."
- 0 train/test leakage, verified; same metric code for every arm.
~672 franglais notes (French prose + English technical terms), 28 topic clusters, a dense semantic
[[wikilink]] graph (≈4.4 links/note, hubs, 0 dangling). Generated by (i) a short natural
conversation distilled into opinionated voice-anchor notes and (ii) a 28-agent Sonnet workflow that
expanded each cluster. Everything is fabricated; it's committed and shareable.
Prereqs: Python 3.10, Node 22, Ollama ≥ 0.30.11 (bundles ROCm v7.1 — auto-detects RDNA4/gfx1201 with zero config; the stock installer's ROCm 6.4.2 silently falls back to CPU), and an AMD GPU with DirectML for Arm B.
python -m pip install -r requirements.txt # CPU stack: baseline, eval, generation, Arm C
# corpus: the committed corpus/ is canonical. To rebuild from the Sonnet output:
python scripts/assemble_corpus.py # -> corpus/vault + manifest + stats
# pairs (anchors) + clean held-out + multi-gold
node scripts/build_corpus.mjs --vault corpus/vault --anchors-only # all verbatim -> train anchors
python scripts/prep_qgen.py # per-cluster inputs for the query workflow
# (held-out + train queries come from a Sonnet workflow over those inputs; processed by:)
python scripts/process_queries.py # -> data/heldout.jsonl + train_aug.jsonl (0 leak)
python scripts/build_multigold.py
# Arm A baseline + eval
python scripts/baseline_embed.py
python scripts/eval.py --preds data/preds_baseline.jsonl --arm A --label e5+bm25
# Arm B (separate DirectML venv) — single GPU consumer, VRAM-guarded
py -3.10 -m venv .venv-dml && .venv-dml/Scripts/python -m pip install -r requirements-directml.txt
.venv-dml/Scripts/python scripts/setup_directml.py # GPU smoke-test
.venv-dml/Scripts/python scripts/train_router.py --size 0.5B --epochs 4 --batch 16 --device dml
.venv-dml/Scripts/python scripts/infer_router.py --size 0.5B --device dml --batch 16 --pause 0.7
python scripts/eval.py --preds data/preds_router_0.5B.jsonl --arm B --label router_0.5B --out results/eval_router_0.5B.json
# (1.5B / 3B: bash scripts/run_all_routers.sh — guarded train->infer->eval, one size after the other)All randomness is seeded from configs/experiment.json (seed: 42); model downloads cache under
.cache/huggingface on the data drive.
EXPERIMENT.md the brief (incl. consolidated precisions)
configs/experiment.json single source of truth (seeds, models, hyperparams)
corpus/ synthetic vault (<slug>.md) + seed notes + generator output
scripts/ gen / assemble / build_corpus / queries / baseline / router / eval / aggregate
data/ train_aug.jsonl, heldout.jsonl, multigold.jsonl, slugs.txt, preds_*.jsonl
results/ metrics dumps + results.json
paper/PAPER.md method, results, distance-to-bar, verdict, threats, related work
Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) © 2026
Emilien Devauchelle. Free to use, share, and adapt for any non-commercial purpose with attribution;
all commercial rights are reserved — contact the author for commercial licensing. See
CITATION.cff to cite this work.