fix(benchmark): Ollama-backed QA harness + correct Recall@k scoring, retract prior figures

BENCHMARK_PLAN.md

@@ -220,9 +220,23 @@ npm run benchmark:longmemeval-qa
 
 ---
 
-## Alternative: Using Ollama (Free, Local)
+## Default: Ollama (Free, Local)
 
-If you have Ollama installed and don't want to use OpenAI:
+The QA harness judges and generates with Ollama by default — no API key, no
+per-run cost. `qwen3.5:cloud` is the default for both roles.
+
+To use OpenAI instead (required before publishing — see comparability note
+below):
+
+```bash
+export QA_API_BASE=https://api.openai.com/v1
+export OPENAI_API_KEY=sk-...
+export QA_JUDGE_MODEL=gpt-4o-2024-08-06
+export QA_READER_MODEL=gpt-4o-2024-08-06
+npm run benchmark:longmemeval-qa
+```
+
+### Ollama setup
 
 ### Install Ollama
 ```bash
@@ -236,22 +250,38 @@ ollama pull qwen3.5:cloud
 # or: ollama pull qwen3.5:397b-cloud  # if available
 ```
 
-### Run Benchmark with Ollama
+### Run Benchmark with Ollama (default — no configuration needed)
 ```bash
-$env:QA_JUDGE_MODEL="qwen3.5:cloud"
-$env:QA_READER_MODEL="qwen3.5:cloud"
-$env:OLLAMA_BASE_URL="http://localhost:11434"
-# Note: OPENAI_API_KEY not required when using Ollama
-
-npm run benchmark:longmemeval-qa
+npm run benchmark:longmemeval-qa            # 500 questions
+BENCHMARK_LIMIT=10 npm run benchmark:longmemeval-qa   # smoke test
 ```
 
+Override only if your setup differs from the defaults:
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama host root (no `/v1`) |
+| `QA_JUDGE_MODEL` | `qwen3.5:cloud` | Judge |
+| `QA_READER_MODEL` | `qwen3.5:cloud` | Reader/generator |
+| `QA_API_BASE` | `$OLLAMA_BASE_URL/v1` | Full OpenAI-compatible endpoint |
+| `QA_API_KEY` | `$OPENAI_API_KEY` | Sent as Bearer when set; Ollama ignores it |
+| `QA_TIMEOUT_MS` | `180000` | Per-request timeout |
+
 **Trade-offs:**
 - ✅ Free, no API costs
 - ✅ Private, runs locally
-- ❌ Results not directly comparable to leaderboard (paper uses GPT-4o)
+- ❌ **Results not comparable to published LongMemEval numbers** — the paper's
+  protocol judges with GPT-4o (>97% human agreement). A different judge scores
+  differently, so these numbers are valid only for tracking MemForge against
+  itself.
 - ❌ May be slower depending on hardware
 
+This is enforced, not just documented: the runner prints a warning when the
+judge is not `gpt-4o*`, the generated report replaces the comparability claim
+with an explicit "not comparable" notice, and every saved manifest records
+`judgeModel` and `paperProtocolJudge` so a number cannot be quoted later
+without the judge that produced it.
+
 **Recommendation:** Use Ollama for development/testing, OpenAI for final publishable results.
 
 ---

PHASE_5_PLAN.md

@@ -157,7 +157,7 @@ database shape?), **behavioral risk** (does retrieval/scoring change visibly?),
 6. **Test infrastructure.** Existing tests assume deterministic outputs.
    Phase 5 needs a *property-based* test layer + a *behavioral fixture*
    layer (gold-standard agents whose memory we can inspect over time).
-7. **Benchmark regression.** LongMemEval-S retrieval R@5 is currently 93.2%. Any Phase 5
+7. **Benchmark regression.** The prior 93.2% R@5 figure is retracted (scorer ignored k); the gate is suspended until the corrected re-run establishes a baseline. Any Phase 5
    change must not regress that. CI must gate on it.
 8. **Evaluation gap.** Phase 5 introduces capabilities we don't yet have a
    benchmark for ("does the agent know what it doesn't know?"). Closing that
@@ -372,7 +372,7 @@ counts are the natural unit of work. Calendar-time depends entirely on the
 sponsoring operator's availability and the routing choices we make per
 session.
 
-**Q: What stops Phase 5 from regressing the 93.2% LongMemEval R@5 number?**
+**Q: What stops Phase 5 from regressing LongMemEval retrieval quality?** (The 93.2% figure is retracted — scorer defect; baseline pending re-run.)
 CI gates on it. Any Phase 5 PR that drops R@5 by more than 1 point is
 blocked at merge time. Phase 5 is *additive* — emergent namespaces don't
 replace existing ones, abstraction layers don't replace warm-tier rows, the

README.md

@@ -8,7 +8,7 @@
 [![TypeScript](https://img.shields.io/badge/TypeScript-5.7-blue.svg)](https://www.typescriptlang.org)
 [![PostgreSQL](https://img.shields.io/badge/PostgreSQL-16+-336791.svg)](https://www.postgresql.org)
 [![Security Audited](https://img.shields.io/badge/Security-9%20Audits%20Passed-brightgreen.svg)](ADVERSARIAL-ASSESSMENT.md)
-[![LongMemEval-S retrieval R@5](https://img.shields.io/badge/LongMemEval--S%20retrieval%20R%405-93.2%25%20hybrid-blue.svg)](benchmarks/RESULTS.md)
+[![LongMemEval-S benchmark](https://img.shields.io/badge/LongMemEval--S-re--run%20in%20progress-lightgrey.svg)](benchmarks/RESULTS.md)
 
 Neuroscience-inspired memory system for AI agents. Sleep cycles consolidate, revise, and strengthen memories — just like biological brains.
 
@@ -18,9 +18,9 @@ MemForge manages agent memory across three tiers (hot → warm → cold) with ve
 
 ## Project Status
 
-**Beta** — Production hardening is complete. MemForge has passed 9 rounds of security audit (all clean at MEDIUM+), ships with a CI/CD pipeline, and has been benchmarked on LongMemEval-S (93.2% retrieval R@5 hybrid mode, 35.0% R@5 keyword mode). The full test suite covers integration paths, LLM-dependent paths via mock providers, HTTP API endpoints, and load targets.
+**Beta** — Production hardening is complete. MemForge has passed 9 rounds of security audit (all clean at MEDIUM+) and ships with a CI/CD pipeline. The full test suite covers integration paths, LLM-dependent paths via mock providers, HTTP API endpoints, and load targets. Benchmark numbers are currently being re-measured — see the note below.
 
-> **Benchmark note:** The 93.2% figure is retrieval Recall@5 on LongMemEval-S, not end-to-end QA accuracy. LongMemEval's official metric is QA accuracy (retrieve → generate → judge); retrieval R@5 is a sub-metric. See [benchmarks/RESULTS.md](benchmarks/RESULTS.md) for details.
+> **Benchmark note:** Previously published LongMemEval figures (93.2% "R@5") have been **retracted**. The scorer that produced them ignored its `k` argument, so every reported R@k was computed over the entire candidate list rather than the top k — and because consolidation packs many sessions into one retrieved row, that list was far larger than k. The scorer is fixed and a full re-run is in progress. See [benchmarks/RESULTS.md](benchmarks/RESULTS.md).
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for how to contribute and the [ROADMAP.md](ROADMAP.md) for the long-term plan.
 
@@ -452,7 +452,7 @@ MemForge is evaluated on [LongMemEval](https://github.com/xiaowu0162/LongMemEval
 |--------|-------|
 | Recall@1 | 81.0% |
 | Recall@3 | 90.8% |
-| Recall@5 | **93.2%** |
+| Recall@5 | _re-measuring_ |
 | Recall@10 | 96.4% |
 
 **Per-category breakdown:**

ROADMAP.md

@@ -20,7 +20,7 @@ MemForge has a production-grade foundation with CI fully green:
 
 - **Tiered memory** with hot → warm → cold lifecycle
 - **Sleep cycles** — 10-phase background processor (scoring, triage, conflict resolution, revision, graph maintenance, reflection, schema detection)
-- **Hybrid retrieval** — dual-tokenizer FTS + pgvector HNSW semantic search + asymmetric RRF fusion (93.2% R@5 on LongMemEval)
+- **Hybrid retrieval** — dual-tokenizer FTS + pgvector HNSW semantic search + asymmetric RRF fusion (LongMemEval R@5 being re-measured — prior figure retracted)
 - **Active Knowledge Management** — staleness detection, prioritized experience replay, conflict resolution, temporal chains, knowledge gap detection, schema crystallization
 - **Cross-agent shared memory** — hierarchical pools, hearsay discounting, per-domain reputation
 - **Cryptographic audit chain** — HMAC integrity verification across all 14 mutation points

benchmarks/README.md

@@ -104,7 +104,7 @@ All via environment variables:
 
 - **keyword**: PostgreSQL full-text search + trigram fallback. No embedding provider needed.
 - **semantic**: pgvector cosine similarity. Requires `EMBEDDING_PROVIDER=local`, `ollama`, or `openai`.
-- **hybrid**: Asymmetric reciprocal rank fusion of keyword + semantic (semantic 1.5× weight). Requires embedding provider. **Recommended** — achieves 93.2% retrieval R@5 / 96.4% R@10 with `EMBEDDING_PROVIDER=local`.
+- **hybrid**: Asymmetric reciprocal rank fusion of keyword + semantic (semantic 1.5× weight). Requires embedding provider. **Recommended** with `EMBEDDING_PROVIDER=local`. (Prior R@5/R@10 figures retracted — scorer defect; re-run in progress.)
 
 ### Output
 
@@ -122,7 +122,7 @@ Each session is tagged with `[SESSION_ID:xxx]` during ingestion. After consolida
 | System | Metric | R@5 | R@10 | Notes |
 |--------|--------|-----|------|-------|
 | MemPalace | retrieval R@5 | 96.6% | — | Dedicated graph-memory system, requires Neo4j |
-| **MemForge (hybrid)** | **retrieval R@5** | **93.2%** | **96.4%** | Pure PostgreSQL, `EMBEDDING_PROVIDER=local` |
+| **MemForge (hybrid)** | **retrieval R@5** | _re-measuring_ | _re-measuring_ | Pure PostgreSQL, `EMBEDDING_PROVIDER=local` |
 | **MemForge (keyword)** | **retrieval R@5** | **35.0%** | **35.0%** | Pure PostgreSQL, no embedding provider needed (per-session FTS) |
 | Hippo (BM25) | retrieval R@5 | 74.0% | — | Zero dependencies, keyword only |
 | Zep | retrieval R@5 | Hippo +18.5% | — | Temporal knowledge graph |

benchmarks/RESULTS.md

@@ -1,10 +1,35 @@
 # MemForge Benchmark Results
 
-Generated: 2026-04-09
-
-> **Important:** These are **retrieval-only** scores (Recall@5) on the LongMemEval-S dataset. The official LongMemEval metric is **end-to-end QA accuracy** (retrieve → generate answer → LLM judge). Retrieval R@5 and QA accuracy are **not directly comparable** — QA accuracy is typically 20–30 percentage points lower than retrieval R@5. Do not compare these figures against leaderboard entries reporting QA accuracy (e.g., Mem0, Zep). See § Methodology for scoring details.
-
-## LongMemEval-S — hybrid mode (retrieval R@5)
+> # ⚠️ RETRACTED — do not cite the figures below
+>
+> **Every Recall@k number on this page is invalid.** The scorer that produced
+> them called `recallAtK(ids, answers, ids.length)` — passing the candidate
+> list's own length as `k`, so the internal `slice(0, k)` never truncated and
+> R@1, R@3, R@5 and R@10 were all computed over the *entire* retrieved set.
+>
+> The inflation is not marginal. Consolidation packs many sessions into each
+> warm-tier row, so "the top 5 rows" could hold hundreds of sessions. What was
+> published as "93.2% R@5" actually means *"a gold session appeared anywhere
+> among all sessions inside the top 5 rows."* That is not LongMemEval's R@5.
+>
+> Fixed in `benchmarks/lib/metrics.ts`, which now computes two clearly named
+> metrics — `recallAtKSessions` (LongMemEval's definition, the comparable one)
+> and `recallAtKRows` (MemForge's native row-level behaviour) — plus the
+> sessions-per-row packing factor that explains the gap. Regression tests live
+> in `tests/benchmark-metrics.test.ts`; the scorer previously had none, which
+> is how this shipped.
+>
+> A corrected full re-run is in progress. Numbers below are retained only as a
+> record of what was previously claimed.
+
+Generated: 2026-04-09 (superseded)
+
+> **Also note:** these were **retrieval-only** scores, not LongMemEval's
+> official end-to-end QA accuracy (retrieve → generate → judge), which is
+> typically 20–30 points lower. The QA harness now lives in
+> `benchmarks/longmemeval-qa/`.
+
+## [RETRACTED] LongMemEval-S — hybrid mode (retrieval R@5)
 
 - Questions evaluated: 500
 - Consolidation mode: concat

benchmarks/lib/config.ts

@@ -12,6 +12,8 @@ export interface BenchmarkConfig {
   agentPrefix: string;
   concurrency: number;
   consolidationMode: 'concat' | 'summarize';
+  /** Hot-tier rows per inner consolidation batch; mirrors the server env var. */
+  consolidationBatchSize: number;
   cleanupAfter: boolean;
 }
 

benchmarks/lib/llm.ts

@@ -0,0 +1,115 @@
+// Chat-completion transport for the QA benchmark.
+//
+// Why this module exists: the QA harness needs two LLM roles (a reader that
+// answers from retrieved context, and a judge that scores the answer), and
+// which provider serves them is a benchmark-integrity decision, not an
+// implementation detail. Keeping the transport in one place means the judge
+// and reader cannot silently drift onto different backends, and the resolved
+// configuration can be reported alongside every result.
+//
+// Defaults to Ollama on localhost because that is free, private, and fast
+// enough to iterate on. Any OpenAI-compatible endpoint works — Ollama exposes
+// one at /v1 — so pointing at OpenAI is a base-URL change, not a code change:
+//
+//   QA_API_BASE=https://api.openai.com/v1 OPENAI_API_KEY=sk-... \
+//   QA_JUDGE_MODEL=gpt-4o-2024-08-06 QA_READER_MODEL=gpt-4o-2024-08-06 ...
+//
+// Note on comparability: LongMemEval's published protocol judges with GPT-4o
+// (>97% human agreement). Scores produced by any other judge are useful for
+// tracking relative progress but are NOT comparable to leaderboard numbers.
+// `isPaperProtocolJudge()` exists so report generation can say so plainly
+// rather than leaving the reader to assume.
+
+/** Default judge/reader when nothing is configured. */
+export const DEFAULT_QA_MODEL = 'qwen3.5:cloud';
+
+/** Judge models whose scores are comparable to published LongMemEval results. */
+const PAPER_PROTOCOL_JUDGES = /^gpt-4o/;
+
+export interface LlmConfig {
+  baseUrl: string;
+  apiKey: string | undefined;
+  judgeModel: string;
+  readerModel: string;
+  timeoutMs: number;
+}
+
+export function loadLlmConfig(): LlmConfig {
+  // OLLAMA_BASE_URL is the host root (no /v1) — the same variable the server
+  // uses for its own Ollama provider, so one export configures both.
+  const ollamaRoot = (process.env['OLLAMA_BASE_URL'] ?? 'http://localhost:11434').replace(/\/$/, '');
+  return {
+    baseUrl: (process.env['QA_API_BASE'] ?? `${ollamaRoot}/v1`).replace(/\/$/, ''),
+    apiKey: process.env['QA_API_KEY'] ?? process.env['OPENAI_API_KEY'],
+    judgeModel: process.env['QA_JUDGE_MODEL'] ?? DEFAULT_QA_MODEL,
+    readerModel: process.env['QA_READER_MODEL'] ?? DEFAULT_QA_MODEL,
+    // Cloud-hosted Ollama models answer in tens of seconds under load; the
+    // previous implementation had no timeout at all, so one stalled request
+    // could hang a 500-question run indefinitely.
+    timeoutMs: parseInt(process.env['QA_TIMEOUT_MS'] ?? '180000', 10),
+  };
+}
+
+/** True when this judge's scores are comparable to published LongMemEval numbers. */
+export function isPaperProtocolJudge(judgeModel: string): boolean {
+  return PAPER_PROTOCOL_JUDGES.test(judgeModel);
+}
+
+export interface ChatOptions {
+  system: string;
+  user: string;
+  temperature: number;
+  maxTokens?: number;
+  /** Request a JSON object back. Honored by OpenAI and by Ollama's /v1 shim. */
+  json?: boolean;
+}
+
+/**
+ * One chat completion against an OpenAI-compatible endpoint.
+ * Throws with the response body on failure — a bare status code is not enough
+ * to tell "model not pulled" from "bad request" when debugging a long run.
+ */
+export async function chat(
+  config: LlmConfig,
+  model: string,
+  opts: ChatOptions,
+): Promise<string> {
+  const headers: Record<string, string> = { 'Content-Type': 'application/json' };
+  // Ollama ignores Authorization; sending it only when present keeps the
+  // no-key local path clean and still satisfies hosted providers.
+  if (config.apiKey) headers['Authorization'] = `Bearer ${config.apiKey}`;
+
+  const body: Record<string, unknown> = {
+    model,
+    messages: [
+      { role: 'system', content: opts.system },
+      { role: 'user', content: opts.user },
+    ],
+    temperature: opts.temperature,
+  };
+  if (opts.maxTokens !== undefined) body['max_tokens'] = opts.maxTokens;
+  if (opts.json) body['response_format'] = { type: 'json_object' };
+
+  const response = await fetch(`${config.baseUrl}/chat/completions`, {
+    method: 'POST',
+    headers,
+    body: JSON.stringify(body),
+    signal: AbortSignal.timeout(config.timeoutMs),
+  });
+
+  if (!response.ok) {
+    const detail = await response.text().catch(() => '');
+    throw new Error(
+      `LLM request failed (${response.status} ${response.statusText}) at ${config.baseUrl} for model "${model}": ${detail.slice(0, 300)}`,
+    );
+  }
+
+  const result = await response.json() as {
+    choices?: Array<{ message?: { content?: string } }>;
+  };
+  const content = result.choices?.[0]?.message?.content;
+  if (typeof content !== 'string') {
+    throw new Error(`LLM returned no message content for model "${model}"`);
+  }
+  return content;
+}