perf: fix keyword search returning ≤1 result and Redis-absent stalls

benchmarks/RESULTS.md

@@ -1,103 +1,108 @@
 # MemForge Benchmark Results
 
-> # ⚠️ RETRACTED — do not cite the figures below
->
-> **Every Recall@k number on this page is invalid.** The scorer that produced
-> them called `recallAtK(ids, answers, ids.length)` — passing the candidate
-> list's own length as `k`, so the internal `slice(0, k)` never truncated and
-> R@1, R@3, R@5 and R@10 were all computed over the *entire* retrieved set.
->
-> The inflation is not marginal. Consolidation packs many sessions into each
-> warm-tier row, so "the top 5 rows" could hold hundreds of sessions. What was
-> published as "93.2% R@5" actually means *"a gold session appeared anywhere
-> among all sessions inside the top 5 rows."* That is not LongMemEval's R@5.
->
-> Fixed in `benchmarks/lib/metrics.ts`, which now computes two clearly named
-> metrics — `recallAtKSessions` (LongMemEval's definition, the comparable one)
-> and `recallAtKRows` (MemForge's native row-level behaviour) — plus the
-> sessions-per-row packing factor that explains the gap. Regression tests live
-> in `tests/benchmark-metrics.test.ts`; the scorer previously had none, which
-> is how this shipped.
->
-> A corrected full re-run is in progress. Numbers below are retained only as a
-> record of what was previously claimed.
-
-Generated: 2026-04-09 (superseded)
+Generated: 2026-07-28
 
-> **Also note:** these were **retrieval-only** scores, not LongMemEval's
-> official end-to-end QA accuracy (retrieve → generate → judge), which is
-> typically 20–30 points lower. The QA harness now lives in
-> `benchmarks/longmemeval-qa/`.
+## LongMemEval — hybrid mode
 
-## [RETRACTED] LongMemEval-S — hybrid mode (retrieval R@5)
+> ## ⚠️ Partial run — not an official result
+>
+> 42 of 500 questions (stratified sample).
+> Reported for development iteration only. Publishable figures require a
+> full 500-question run; cite nothing from this page until then.
 
-- Questions evaluated: 500
+- Questions evaluated: 42 of 500 (stratified sample)
 - Consolidation mode: concat
-- Timestamp: 2026-04-09T08:05:52.269Z
+- Timestamp: 2026-07-28T01:21:09.050Z
 
 ### Retrieval Quality
 
-| Metric | Score |
-|--------|-------|
-| Recall@1 | 81.0% |
-| Recall@3 | 90.8% |
-| Recall@5 | 93.2% |
-| Recall@10 | 96.4% |
+> **Recall@k (sessions)** is the comparable metric: a hit means a gold
+> session is among the first k distinct sessions by rank — LongMemEval's
+> definition. **Recall@k (rows)** counts a hit anywhere inside the top-k
+> retrieved rows; because consolidation packs multiple sessions per row it
+> is strictly more generous and is NOT comparable to published figures.
 
-**Baselines:** Hippo 74.0% R@5 (BM25 keyword), Zep +18.5% over full-context
+| Metric | Sessions (comparable) | Rows (native) |
+|--------|----------------------|---------------|
+| Recall@1 | 69.0% | 69.0% |
+| Recall@3 | 85.7% | 85.7% |
+| Recall@5 | 92.9% | 92.9% |
+| Recall@10 | 92.9% | 92.9% |
+
+Sessions packed per retrieved row: **1.0**
+(1.0 means rows and sessions are 1:1 and the two columns converge.)
+
+**Baselines (compare against the Sessions column only):** Hippo 74.0% R@5
+(BM25 keyword), Zep +18.5% over full-context
 
 ### Per-Category Breakdown
 
-| Category | Count | R@1 | R@3 | R@5 | R@10 |
+| Category | Count | R@1 (sessions) | R@3 (sessions) | R@5 (sessions) | R@10 (sessions) |
 |----------|-------|------|------|------|------|
-| knowledge-update | 78 | 93.6% | 97.4% | 97.4% | 100.0% |
-| multi-session | 133 | 86.5% | 94.0% | 96.2% | 98.5% |
-| single-session-assistant | 56 | 92.9% | 98.2% | 100.0% | 100.0% |
-| single-session-preference | 30 | 43.3% | 66.7% | 80.0% | 93.3% |
-| single-session-user | 70 | 74.3% | 84.3% | 87.1% | 90.0% |
-| temporal-reasoning | 133 | 75.2% | 89.5% | 91.0% | 94.7% |
+| knowledge-update | 7 | 85.7% | 100.0% | 100.0% | 100.0% |
+| multi-session | 7 | 85.7% | 85.7% | 100.0% | 100.0% |
+| single-session-assistant | 7 | 85.7% | 100.0% | 100.0% | 100.0% |
+| single-session-preference | 7 | 42.9% | 71.4% | 85.7% | 85.7% |
+| single-session-user | 7 | 57.1% | 71.4% | 85.7% | 85.7% |
+| temporal-reasoning | 7 | 57.1% | 85.7% | 85.7% | 85.7% |
 
 ### Latency
 
 | Operation | p50 | p95 | Mean |
 |-----------|-----|-----|------|
-| Query | 45ms | 77ms | 48ms |
-| Ingest (per question) | 23.1s | 35.8s | 23.1s |
+| Query | 27ms | 52ms | 31ms |
+| Ingest (per question) | 5.2s | 10.7s | 5.4s |
 
 ## LongMemEval — keyword mode
 
-- Questions evaluated: 500
+> ## ⚠️ Partial run — not an official result
+>
+> 42 of 500 questions (stratified sample).
+> Reported for development iteration only. Publishable figures require a
+> full 500-question run; cite nothing from this page until then.
+
+- Questions evaluated: 42 of 500 (stratified sample)
 - Consolidation mode: concat
-- Timestamp: 2026-04-09T08:05:52.272Z
+- Timestamp: 2026-07-28T01:21:09.051Z
 
 ### Retrieval Quality
 
-| Metric | Score |
-|--------|-------|
-| Recall@1 | 33.4% |
-| Recall@3 | 34.6% |
-| Recall@5 | 35.0% |
-| Recall@10 | 35.0% |
+> **Recall@k (sessions)** is the comparable metric: a hit means a gold
+> session is among the first k distinct sessions by rank — LongMemEval's
+> definition. **Recall@k (rows)** counts a hit anywhere inside the top-k
+> retrieved rows; because consolidation packs multiple sessions per row it
+> is strictly more generous and is NOT comparable to published figures.
+
+| Metric | Sessions (comparable) | Rows (native) |
+|--------|----------------------|---------------|
+| Recall@1 | 69.0% | 69.0% |
+| Recall@3 | 83.3% | 83.3% |
+| Recall@5 | 92.9% | 92.9% |
+| Recall@10 | 95.2% | 95.2% |
+
+Sessions packed per retrieved row: **1.0**
+(1.0 means rows and sessions are 1:1 and the two columns converge.)
 
-**Baselines:** Hippo 74.0% R@5 (BM25 keyword), Zep +18.5% over full-context
+**Baselines (compare against the Sessions column only):** Hippo 74.0% R@5
+(BM25 keyword), Zep +18.5% over full-context
 
 ### Per-Category Breakdown
 
-| Category | Count | R@1 | R@3 | R@5 | R@10 |
+| Category | Count | R@1 (sessions) | R@3 (sessions) | R@5 (sessions) | R@10 (sessions) |
 |----------|-------|------|------|------|------|
-| knowledge-update | 78 | 56.4% | 56.4% | 56.4% | 56.4% |
-| multi-session | 133 | 29.3% | 30.1% | 30.1% | 30.1% |
-| single-session-assistant | 56 | 19.6% | 23.2% | 25.0% | 25.0% |
-| single-session-preference | 30 | 6.7% | 10.0% | 10.0% | 10.0% |
-| single-session-user | 70 | 57.1% | 58.6% | 58.6% | 58.6% |
-| temporal-reasoning | 133 | 23.3% | 24.1% | 24.8% | 24.8% |
+| knowledge-update | 7 | 85.7% | 100.0% | 100.0% | 100.0% |
+| multi-session | 7 | 85.7% | 85.7% | 100.0% | 100.0% |
+| single-session-assistant | 7 | 57.1% | 100.0% | 100.0% | 100.0% |
+| single-session-preference | 7 | 42.9% | 42.9% | 71.4% | 85.7% |
+| single-session-user | 7 | 71.4% | 85.7% | 100.0% | 100.0% |
+| temporal-reasoning | 7 | 71.4% | 85.7% | 85.7% | 85.7% |
 
 ### Latency
 
 | Operation | p50 | p95 | Mean |
 |-----------|-----|-----|------|
-| Query | 14ms | 26ms | 14ms |
-| Ingest (per question) | 23.1s | 35.8s | 23.1s |
+| Query | 9ms | 40ms | 12ms |
+| Ingest (per question) | 5.2s | 10.7s | 5.4s |
 
 ---
 

benchmarks/lib/sample.ts

@@ -0,0 +1,79 @@
+// Subset selection for limited benchmark runs.
+//
+// LongMemEval-S stores its 500 instances in contiguous blocks by question
+// type: single-session-user ×70, multi-session ×62, single-session-preference
+// ×30, multi-session ×71, temporal-reasoning ×133, knowledge-update ×78,
+// single-session-assistant ×56.
+//
+// A plain `slice(offset, offset + limit)` therefore samples one or two
+// categories and nothing else — `BENCHMARK_LIMIT=50` returns 50
+// single-session-user questions. Any number produced that way describes one
+// question type while looking like an overall score, and tuning retrieval
+// against it optimises for whichever category happens to sit at the offset.
+//
+// Stratified selection round-robins across the types present, so a 50-question
+// run covers every category in proportion. It is deterministic — no RNG, and
+// original order is preserved within each type — so runs stay comparable
+// across code changes, which is the whole point of a regression benchmark.
+
+export type SampleStrategy = 'stratified' | 'sequential';
+
+export function resolveStrategy(raw: string | undefined): SampleStrategy {
+  return raw === 'sequential' ? 'sequential' : 'stratified';
+}
+
+/**
+ * Pick `limit` instances starting at `offset`.
+ *
+ * `sequential` reproduces the original slice — use it to re-measure an exact
+ * historical run, or when evaluating the full dataset where the distinction
+ * does not apply.
+ *
+ * `stratified` (default) takes instances round-robin by `question_type`, which
+ * for limit >= number-of-types yields a representative mix. Returned items
+ * carry their original dataset index so results stay traceable.
+ */
+export function selectSubset<T extends { question_type?: string }>(
+  dataset: T[],
+  limit: number,
+  offset: number,
+  strategy: SampleStrategy,
+): Array<{ instance: T; datasetIndex: number }> {
+  const indexed = dataset.map((instance, datasetIndex) => ({ instance, datasetIndex }));
+
+  if (strategy === 'sequential' || limit >= dataset.length) {
+    return indexed.slice(offset, offset + limit);
+  }
+
+  // Group by type, preserving dataset order within each group.
+  const byType = new Map<string, Array<{ instance: T; datasetIndex: number }>>();
+  for (const entry of indexed) {
+    const type = entry.instance.question_type ?? 'unknown';
+    const bucket = byType.get(type) ?? [];
+    bucket.push(entry);
+    byType.set(type, bucket);
+  }
+
+  // Offset advances the starting cursor within every bucket, so successive
+  // offsets walk disjoint questions rather than re-drawing the same ones.
+  const types = [...byType.keys()].sort();
+  const cursors = new Map(types.map((t) => [t, offset]));
+
+  const picked: Array<{ instance: T; datasetIndex: number }> = [];
+  let exhausted = false;
+  while (picked.length < limit && !exhausted) {
+    exhausted = true;
+    for (const type of types) {
+      if (picked.length >= limit) break;
+      const bucket = byType.get(type)!;
+      const cursor = cursors.get(type)!;
+      if (cursor < bucket.length) {
+        picked.push(bucket[cursor]!);
+        cursors.set(type, cursor + 1);
+        exhausted = false;
+      }
+    }
+  }
+
+  return picked;
+}

benchmarks/longmemeval/ingest.ts

@@ -8,6 +8,7 @@ import { join } from 'node:path';
 import { performance } from 'node:perf_hooks';
 import { loadConfig, type BenchmarkConfig } from '../lib/config.js';
 import { createLimiter } from '../lib/concurrency.js';
+import { selectSubset, resolveStrategy } from '../lib/sample.js';
 import type { LongMemEvalInstance, IngestManifest } from './types.js';
 
 // Dynamic import to avoid module-level side effects
@@ -88,8 +89,16 @@ export async function main(configOverride?: BenchmarkConfig): Promise<IngestMani
   // Load dataset
   const dataFile = join(config.datasetDir, 'longmemeval_s.json');
   const dataset = JSON.parse(readFileSync(dataFile, 'utf-8')) as LongMemEvalInstance[];
-  const questions = dataset.slice(config.questionOffset, config.questionOffset + config.questionLimit);
-  console.log(`Loaded ${dataset.length} instances, processing ${questions.length}`);
+  const strategy = resolveStrategy(process.env['BENCHMARK_SAMPLE']);
+  const selected = selectSubset(dataset, config.questionLimit, config.questionOffset, strategy);
+  const questions = selected.map((s) => s.instance);
+  const typeCounts = questions.reduce<Record<string, number>>((acc, q) => {
+    const t = q.question_type ?? 'unknown';
+    acc[t] = (acc[t] ?? 0) + 1;
+    return acc;
+  }, {});
+  console.log(`Loaded ${dataset.length} instances, processing ${questions.length} (sampling: ${strategy})`);
+  console.log(`  type mix: ${Object.entries(typeCounts).map(([t, n]) => `${t}=${n}`).join(', ')}`);
 
   // Health check
   const client = await createClient(config);
@@ -107,7 +116,7 @@ export async function main(configOverride?: BenchmarkConfig): Promise<IngestMani
   const totalStart = performance.now();
 
   const promises = questions.map((instance, i) => {
-    const questionIndex = config.questionOffset + i;
+    const questionIndex = selected[i]!.datasetIndex;
     return limit(async () => {
       // Retry up to 2 times on transient failures (timeout, connection reset)
       for (let attempt = 0; attempt < 3; attempt++) {

benchmarks/longmemeval/report.ts

@@ -33,7 +33,21 @@ function generateMarkdown(reports: BenchmarkReport[]): string {
   for (const report of reports) {
     lines.push(`## LongMemEval — ${report.queryMode} mode`);
     lines.push('');
-    lines.push(`- Questions evaluated: ${report.questionsEvaluated}`);
+    // A partial run must never read like the official measurement. The
+    // generator overwrites RESULTS.md in place, so without this a 20-question
+    // smoke test silently replaces the published page.
+    const FULL_DATASET = 500;
+    const isFullRun = report.questionsEvaluated >= FULL_DATASET;
+    const sampling = process.env['BENCHMARK_SAMPLE'] === 'sequential' ? 'sequential' : 'stratified';
+    if (!isFullRun) {
+      lines.push(`> ## ⚠️ Partial run — not an official result`);
+      lines.push('>');
+      lines.push(`> ${report.questionsEvaluated} of ${FULL_DATASET} questions (${sampling} sample).`);
+      lines.push('> Reported for development iteration only. Publishable figures require a');
+      lines.push('> full 500-question run; cite nothing from this page until then.');
+      lines.push('');
+    }
+    lines.push(`- Questions evaluated: ${report.questionsEvaluated}${isFullRun ? '' : ` of ${FULL_DATASET} (${sampling} sample)`}`);
     lines.push(`- Consolidation mode: ${report.consolidationMode}`);
     lines.push(`- Timestamp: ${report.timestamp}`);
     lines.push('');

package.json

@@ -45,7 +45,7 @@
     "test:dreams-compat": "node --import tsx/esm --test tests/dreams-compat.test.ts",
     "test:dreams-anthropic": "node --import tsx/esm --test tests/dreams-anthropic.test.ts",
     "test:dreams-bridge": "node --import tsx/esm --test tests/dreams-bridge.test.ts",
-    "test": "node --import tsx/esm --test --test-concurrency=1 tests/integration.test.ts tests/llm-paths.test.ts tests/http-api.test.ts tests/cache.test.ts tests/embedding-migration.test.ts tests/outcome-revision.test.ts tests/reflection-revision.test.ts tests/selective-forgetting.test.ts tests/multi-device.test.ts tests/dream-runs.test.ts tests/dreams-compat.test.ts tests/dreams-anthropic.test.ts tests/dreams-bridge.test.ts tests/sentiment-tagging.test.ts tests/adaptive-sleep.test.ts tests/epistemic-confidence.test.ts tests/explainable-memory.test.ts tests/causal-graph.test.ts tests/abstractions.test.ts tests/bootstrap.test.ts tests/contested-conflicts.test.ts tests/benchmark-metrics.test.ts",
+    "test": "node --import tsx/esm --test --test-concurrency=1 tests/integration.test.ts tests/llm-paths.test.ts tests/http-api.test.ts tests/cache.test.ts tests/embedding-migration.test.ts tests/outcome-revision.test.ts tests/reflection-revision.test.ts tests/selective-forgetting.test.ts tests/multi-device.test.ts tests/dream-runs.test.ts tests/dreams-compat.test.ts tests/dreams-anthropic.test.ts tests/dreams-bridge.test.ts tests/sentiment-tagging.test.ts tests/adaptive-sleep.test.ts tests/epistemic-confidence.test.ts tests/explainable-memory.test.ts tests/causal-graph.test.ts tests/abstractions.test.ts tests/bootstrap.test.ts tests/contested-conflicts.test.ts tests/benchmark-metrics.test.ts tests/cache-degradation.test.ts",
     "test:multi-device": "node --import tsx/esm --test tests/multi-device.test.ts",
     "test:sentiment-tagging": "node --import tsx/esm --test tests/sentiment-tagging.test.ts",
     "test:adaptive-sleep": "node --import tsx/esm --test tests/adaptive-sleep.test.ts",
@@ -56,6 +56,7 @@
     "test:bootstrap": "node --import tsx/esm --test tests/bootstrap.test.ts",
     "test:contested-conflicts": "node --import tsx/esm --test tests/contested-conflicts.test.ts",
     "test:benchmark-metrics": "node --import tsx/esm --test tests/benchmark-metrics.test.ts",
+    "test:cache-degradation": "node --import tsx/esm --test tests/cache-degradation.test.ts",
     "benchmark:longmemeval": "node --import tsx/esm benchmarks/longmemeval/run.ts",
     "benchmark:download": "node --import tsx/esm benchmarks/longmemeval/download.ts",
     "benchmark:ingest": "node --import tsx/esm benchmarks/longmemeval/ingest.ts",

src/cache.ts

@@ -62,11 +62,41 @@ const counters: CacheCounters = { hits: 0, misses: 0, sets: 0, invalidations: 0,
 let redisClient: RedisClientType | null = null;
 let connectionPromise: Promise<RedisClientType | null> | null = null;
 
+/**
+ * Circuit breaker for an unreachable Redis.
+ *
+ * A failed connect costs ~7.5s: connectTimeout plus the reconnect backoff
+ * ladder before the client gives up. Without this, that price was paid on
+ * *every* cache read, because the failed attempt left no memory of itself —
+ * so a deployment with Redis simply absent (a documented, supported
+ * configuration) had multi-second latency on every cached endpoint while
+ * appearing to "degrade gracefully".
+ *
+ * After a failure we skip Redis outright until the cooldown expires, then
+ * allow exactly one probe. Reads fall through to the database, which is the
+ * intended degraded behaviour — just immediately instead of eventually.
+ */
+const CONNECT_FAILURE_COOLDOWN_MS = parseInt(
+  process.env['REDIS_RETRY_COOLDOWN_MS'] ?? '30000',
+  10,
+);
+let connectFailedAt = 0;
+
+/** Test seam: forget the breaker state so a suite can exercise both paths. */
+export function resetRedisCircuitBreaker(): void {
+  connectFailedAt = 0;
+}
+
 // ─── Connection ───────────────────────────────────────────────────────────────
 
 export async function getRedis(): Promise<RedisClientType | null> {
   if (redisClient?.isOpen) return redisClient;
 
+  // Breaker open — fail fast rather than re-paying the connect timeout.
+  if (connectFailedAt !== 0 && Date.now() - connectFailedAt < CONNECT_FAILURE_COOLDOWN_MS) {
+    return null;
+  }
+
   // Coalesce concurrent connection attempts
   if (connectionPromise) return connectionPromise;
 
@@ -103,9 +133,14 @@ export async function getRedis(): Promise<RedisClientType | null> {
       const safeUrl = url.replace(/:\/\/[^@]*@/, '://*:*@');
       log.info({ url: safeUrl }, 'Redis connected');
       redisClient = client;
+      connectFailedAt = 0; // recovered — close the breaker
       return client;
     } catch (err) {
-      log.error({ err }, 'Redis connection failed — operating without cache');
+      connectFailedAt = Date.now();
+      log.error(
+        { err, cooldownMs: CONNECT_FAILURE_COOLDOWN_MS },
+        'Redis connection failed — operating without cache; suppressing retries for the cooldown',
+      );
       return null;
     } finally {
       connectionPromise = null;