diff --git a/BENCHMARK_PLAN.md b/BENCHMARK_PLAN.md index a0a39ed..90bf5c3 100644 --- a/BENCHMARK_PLAN.md +++ b/BENCHMARK_PLAN.md @@ -220,9 +220,23 @@ npm run benchmark:longmemeval-qa --- -## Alternative: Using Ollama (Free, Local) +## Default: Ollama (Free, Local) -If you have Ollama installed and don't want to use OpenAI: +The QA harness judges and generates with Ollama by default — no API key, no +per-run cost. `qwen3.5:cloud` is the default for both roles. + +To use OpenAI instead (required before publishing — see comparability note +below): + +```bash +export QA_API_BASE=https://api.openai.com/v1 +export OPENAI_API_KEY=sk-... +export QA_JUDGE_MODEL=gpt-4o-2024-08-06 +export QA_READER_MODEL=gpt-4o-2024-08-06 +npm run benchmark:longmemeval-qa +``` + +### Ollama setup ### Install Ollama ```bash @@ -236,22 +250,38 @@ ollama pull qwen3.5:cloud # or: ollama pull qwen3.5:397b-cloud # if available ``` -### Run Benchmark with Ollama +### Run Benchmark with Ollama (default — no configuration needed) ```bash -$env:QA_JUDGE_MODEL="qwen3.5:cloud" -$env:QA_READER_MODEL="qwen3.5:cloud" -$env:OLLAMA_BASE_URL="http://localhost:11434" -# Note: OPENAI_API_KEY not required when using Ollama - -npm run benchmark:longmemeval-qa +npm run benchmark:longmemeval-qa # 500 questions +BENCHMARK_LIMIT=10 npm run benchmark:longmemeval-qa # smoke test ``` +Override only if your setup differs from the defaults: + +| Variable | Default | Purpose | +|----------|---------|---------| +| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama host root (no `/v1`) | +| `QA_JUDGE_MODEL` | `qwen3.5:cloud` | Judge | +| `QA_READER_MODEL` | `qwen3.5:cloud` | Reader/generator | +| `QA_API_BASE` | `$OLLAMA_BASE_URL/v1` | Full OpenAI-compatible endpoint | +| `QA_API_KEY` | `$OPENAI_API_KEY` | Sent as Bearer when set; Ollama ignores it | +| `QA_TIMEOUT_MS` | `180000` | Per-request timeout | + **Trade-offs:** - ✅ Free, no API costs - ✅ Private, runs locally -- ❌ Results not directly comparable to leaderboard (paper uses GPT-4o) +- ❌ **Results not comparable to published LongMemEval numbers** — the paper's + protocol judges with GPT-4o (>97% human agreement). A different judge scores + differently, so these numbers are valid only for tracking MemForge against + itself. - ❌ May be slower depending on hardware +This is enforced, not just documented: the runner prints a warning when the +judge is not `gpt-4o*`, the generated report replaces the comparability claim +with an explicit "not comparable" notice, and every saved manifest records +`judgeModel` and `paperProtocolJudge` so a number cannot be quoted later +without the judge that produced it. + **Recommendation:** Use Ollama for development/testing, OpenAI for final publishable results. --- diff --git a/PHASE_5_PLAN.md b/PHASE_5_PLAN.md index 408d445..b9dca26 100644 --- a/PHASE_5_PLAN.md +++ b/PHASE_5_PLAN.md @@ -157,7 +157,7 @@ database shape?), **behavioral risk** (does retrieval/scoring change visibly?), 6. **Test infrastructure.** Existing tests assume deterministic outputs. Phase 5 needs a *property-based* test layer + a *behavioral fixture* layer (gold-standard agents whose memory we can inspect over time). -7. **Benchmark regression.** LongMemEval-S retrieval R@5 is currently 93.2%. Any Phase 5 +7. **Benchmark regression.** The prior 93.2% R@5 figure is retracted (scorer ignored k); the gate is suspended until the corrected re-run establishes a baseline. Any Phase 5 change must not regress that. CI must gate on it. 8. **Evaluation gap.** Phase 5 introduces capabilities we don't yet have a benchmark for ("does the agent know what it doesn't know?"). Closing that @@ -372,7 +372,7 @@ counts are the natural unit of work. Calendar-time depends entirely on the sponsoring operator's availability and the routing choices we make per session. -**Q: What stops Phase 5 from regressing the 93.2% LongMemEval R@5 number?** +**Q: What stops Phase 5 from regressing LongMemEval retrieval quality?** (The 93.2% figure is retracted — scorer defect; baseline pending re-run.) CI gates on it. Any Phase 5 PR that drops R@5 by more than 1 point is blocked at merge time. Phase 5 is *additive* — emergent namespaces don't replace existing ones, abstraction layers don't replace warm-tier rows, the diff --git a/README.md b/README.md index 1162cf9..68c4cb3 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ [![TypeScript](https://img.shields.io/badge/TypeScript-5.7-blue.svg)](https://www.typescriptlang.org) [![PostgreSQL](https://img.shields.io/badge/PostgreSQL-16+-336791.svg)](https://www.postgresql.org) [![Security Audited](https://img.shields.io/badge/Security-9%20Audits%20Passed-brightgreen.svg)](ADVERSARIAL-ASSESSMENT.md) -[![LongMemEval-S retrieval R@5](https://img.shields.io/badge/LongMemEval--S%20retrieval%20R%405-93.2%25%20hybrid-blue.svg)](benchmarks/RESULTS.md) +[![LongMemEval-S benchmark](https://img.shields.io/badge/LongMemEval--S-re--run%20in%20progress-lightgrey.svg)](benchmarks/RESULTS.md) Neuroscience-inspired memory system for AI agents. Sleep cycles consolidate, revise, and strengthen memories — just like biological brains. @@ -18,9 +18,9 @@ MemForge manages agent memory across three tiers (hot → warm → cold) with ve ## Project Status -**Beta** — Production hardening is complete. MemForge has passed 9 rounds of security audit (all clean at MEDIUM+), ships with a CI/CD pipeline, and has been benchmarked on LongMemEval-S (93.2% retrieval R@5 hybrid mode, 35.0% R@5 keyword mode). The full test suite covers integration paths, LLM-dependent paths via mock providers, HTTP API endpoints, and load targets. +**Beta** — Production hardening is complete. MemForge has passed 9 rounds of security audit (all clean at MEDIUM+) and ships with a CI/CD pipeline. The full test suite covers integration paths, LLM-dependent paths via mock providers, HTTP API endpoints, and load targets. Benchmark numbers are currently being re-measured — see the note below. -> **Benchmark note:** The 93.2% figure is retrieval Recall@5 on LongMemEval-S, not end-to-end QA accuracy. LongMemEval's official metric is QA accuracy (retrieve → generate → judge); retrieval R@5 is a sub-metric. See [benchmarks/RESULTS.md](benchmarks/RESULTS.md) for details. +> **Benchmark note:** Previously published LongMemEval figures (93.2% "R@5") have been **retracted**. The scorer that produced them ignored its `k` argument, so every reported R@k was computed over the entire candidate list rather than the top k — and because consolidation packs many sessions into one retrieved row, that list was far larger than k. The scorer is fixed and a full re-run is in progress. See [benchmarks/RESULTS.md](benchmarks/RESULTS.md). See [CONTRIBUTING.md](CONTRIBUTING.md) for how to contribute and the [ROADMAP.md](ROADMAP.md) for the long-term plan. @@ -452,7 +452,7 @@ MemForge is evaluated on [LongMemEval](https://github.com/xiaowu0162/LongMemEval |--------|-------| | Recall@1 | 81.0% | | Recall@3 | 90.8% | -| Recall@5 | **93.2%** | +| Recall@5 | _re-measuring_ | | Recall@10 | 96.4% | **Per-category breakdown:** diff --git a/ROADMAP.md b/ROADMAP.md index 71f6db8..9c80fff 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -20,7 +20,7 @@ MemForge has a production-grade foundation with CI fully green: - **Tiered memory** with hot → warm → cold lifecycle - **Sleep cycles** — 10-phase background processor (scoring, triage, conflict resolution, revision, graph maintenance, reflection, schema detection) -- **Hybrid retrieval** — dual-tokenizer FTS + pgvector HNSW semantic search + asymmetric RRF fusion (93.2% R@5 on LongMemEval) +- **Hybrid retrieval** — dual-tokenizer FTS + pgvector HNSW semantic search + asymmetric RRF fusion (LongMemEval R@5 being re-measured — prior figure retracted) - **Active Knowledge Management** — staleness detection, prioritized experience replay, conflict resolution, temporal chains, knowledge gap detection, schema crystallization - **Cross-agent shared memory** — hierarchical pools, hearsay discounting, per-domain reputation - **Cryptographic audit chain** — HMAC integrity verification across all 14 mutation points diff --git a/benchmarks/README.md b/benchmarks/README.md index 20b8702..8b1993e 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -104,7 +104,7 @@ All via environment variables: - **keyword**: PostgreSQL full-text search + trigram fallback. No embedding provider needed. - **semantic**: pgvector cosine similarity. Requires `EMBEDDING_PROVIDER=local`, `ollama`, or `openai`. -- **hybrid**: Asymmetric reciprocal rank fusion of keyword + semantic (semantic 1.5× weight). Requires embedding provider. **Recommended** — achieves 93.2% retrieval R@5 / 96.4% R@10 with `EMBEDDING_PROVIDER=local`. +- **hybrid**: Asymmetric reciprocal rank fusion of keyword + semantic (semantic 1.5× weight). Requires embedding provider. **Recommended** with `EMBEDDING_PROVIDER=local`. (Prior R@5/R@10 figures retracted — scorer defect; re-run in progress.) ### Output @@ -122,7 +122,7 @@ Each session is tagged with `[SESSION_ID:xxx]` during ingestion. After consolida | System | Metric | R@5 | R@10 | Notes | |--------|--------|-----|------|-------| | MemPalace | retrieval R@5 | 96.6% | — | Dedicated graph-memory system, requires Neo4j | -| **MemForge (hybrid)** | **retrieval R@5** | **93.2%** | **96.4%** | Pure PostgreSQL, `EMBEDDING_PROVIDER=local` | +| **MemForge (hybrid)** | **retrieval R@5** | _re-measuring_ | _re-measuring_ | Pure PostgreSQL, `EMBEDDING_PROVIDER=local` | | **MemForge (keyword)** | **retrieval R@5** | **35.0%** | **35.0%** | Pure PostgreSQL, no embedding provider needed (per-session FTS) | | Hippo (BM25) | retrieval R@5 | 74.0% | — | Zero dependencies, keyword only | | Zep | retrieval R@5 | Hippo +18.5% | — | Temporal knowledge graph | diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md index 30e7363..ae99dd3 100644 --- a/benchmarks/RESULTS.md +++ b/benchmarks/RESULTS.md @@ -1,10 +1,35 @@ # MemForge Benchmark Results -Generated: 2026-04-09 - -> **Important:** These are **retrieval-only** scores (Recall@5) on the LongMemEval-S dataset. The official LongMemEval metric is **end-to-end QA accuracy** (retrieve → generate answer → LLM judge). Retrieval R@5 and QA accuracy are **not directly comparable** — QA accuracy is typically 20–30 percentage points lower than retrieval R@5. Do not compare these figures against leaderboard entries reporting QA accuracy (e.g., Mem0, Zep). See § Methodology for scoring details. - -## LongMemEval-S — hybrid mode (retrieval R@5) +> # ⚠️ RETRACTED — do not cite the figures below +> +> **Every Recall@k number on this page is invalid.** The scorer that produced +> them called `recallAtK(ids, answers, ids.length)` — passing the candidate +> list's own length as `k`, so the internal `slice(0, k)` never truncated and +> R@1, R@3, R@5 and R@10 were all computed over the *entire* retrieved set. +> +> The inflation is not marginal. Consolidation packs many sessions into each +> warm-tier row, so "the top 5 rows" could hold hundreds of sessions. What was +> published as "93.2% R@5" actually means *"a gold session appeared anywhere +> among all sessions inside the top 5 rows."* That is not LongMemEval's R@5. +> +> Fixed in `benchmarks/lib/metrics.ts`, which now computes two clearly named +> metrics — `recallAtKSessions` (LongMemEval's definition, the comparable one) +> and `recallAtKRows` (MemForge's native row-level behaviour) — plus the +> sessions-per-row packing factor that explains the gap. Regression tests live +> in `tests/benchmark-metrics.test.ts`; the scorer previously had none, which +> is how this shipped. +> +> A corrected full re-run is in progress. Numbers below are retained only as a +> record of what was previously claimed. + +Generated: 2026-04-09 (superseded) + +> **Also note:** these were **retrieval-only** scores, not LongMemEval's +> official end-to-end QA accuracy (retrieve → generate → judge), which is +> typically 20–30 points lower. The QA harness now lives in +> `benchmarks/longmemeval-qa/`. + +## [RETRACTED] LongMemEval-S — hybrid mode (retrieval R@5) - Questions evaluated: 500 - Consolidation mode: concat diff --git a/benchmarks/lib/config.ts b/benchmarks/lib/config.ts index a525f59..ea934c8 100644 --- a/benchmarks/lib/config.ts +++ b/benchmarks/lib/config.ts @@ -12,6 +12,8 @@ export interface BenchmarkConfig { agentPrefix: string; concurrency: number; consolidationMode: 'concat' | 'summarize'; + /** Hot-tier rows per inner consolidation batch; mirrors the server env var. */ + consolidationBatchSize: number; cleanupAfter: boolean; } diff --git a/benchmarks/lib/llm.ts b/benchmarks/lib/llm.ts new file mode 100644 index 0000000..da6b3c8 --- /dev/null +++ b/benchmarks/lib/llm.ts @@ -0,0 +1,115 @@ +// Chat-completion transport for the QA benchmark. +// +// Why this module exists: the QA harness needs two LLM roles (a reader that +// answers from retrieved context, and a judge that scores the answer), and +// which provider serves them is a benchmark-integrity decision, not an +// implementation detail. Keeping the transport in one place means the judge +// and reader cannot silently drift onto different backends, and the resolved +// configuration can be reported alongside every result. +// +// Defaults to Ollama on localhost because that is free, private, and fast +// enough to iterate on. Any OpenAI-compatible endpoint works — Ollama exposes +// one at /v1 — so pointing at OpenAI is a base-URL change, not a code change: +// +// QA_API_BASE=https://api.openai.com/v1 OPENAI_API_KEY=sk-... \ +// QA_JUDGE_MODEL=gpt-4o-2024-08-06 QA_READER_MODEL=gpt-4o-2024-08-06 ... +// +// Note on comparability: LongMemEval's published protocol judges with GPT-4o +// (>97% human agreement). Scores produced by any other judge are useful for +// tracking relative progress but are NOT comparable to leaderboard numbers. +// `isPaperProtocolJudge()` exists so report generation can say so plainly +// rather than leaving the reader to assume. + +/** Default judge/reader when nothing is configured. */ +export const DEFAULT_QA_MODEL = 'qwen3.5:cloud'; + +/** Judge models whose scores are comparable to published LongMemEval results. */ +const PAPER_PROTOCOL_JUDGES = /^gpt-4o/; + +export interface LlmConfig { + baseUrl: string; + apiKey: string | undefined; + judgeModel: string; + readerModel: string; + timeoutMs: number; +} + +export function loadLlmConfig(): LlmConfig { + // OLLAMA_BASE_URL is the host root (no /v1) — the same variable the server + // uses for its own Ollama provider, so one export configures both. + const ollamaRoot = (process.env['OLLAMA_BASE_URL'] ?? 'http://localhost:11434').replace(/\/$/, ''); + return { + baseUrl: (process.env['QA_API_BASE'] ?? `${ollamaRoot}/v1`).replace(/\/$/, ''), + apiKey: process.env['QA_API_KEY'] ?? process.env['OPENAI_API_KEY'], + judgeModel: process.env['QA_JUDGE_MODEL'] ?? DEFAULT_QA_MODEL, + readerModel: process.env['QA_READER_MODEL'] ?? DEFAULT_QA_MODEL, + // Cloud-hosted Ollama models answer in tens of seconds under load; the + // previous implementation had no timeout at all, so one stalled request + // could hang a 500-question run indefinitely. + timeoutMs: parseInt(process.env['QA_TIMEOUT_MS'] ?? '180000', 10), + }; +} + +/** True when this judge's scores are comparable to published LongMemEval numbers. */ +export function isPaperProtocolJudge(judgeModel: string): boolean { + return PAPER_PROTOCOL_JUDGES.test(judgeModel); +} + +export interface ChatOptions { + system: string; + user: string; + temperature: number; + maxTokens?: number; + /** Request a JSON object back. Honored by OpenAI and by Ollama's /v1 shim. */ + json?: boolean; +} + +/** + * One chat completion against an OpenAI-compatible endpoint. + * Throws with the response body on failure — a bare status code is not enough + * to tell "model not pulled" from "bad request" when debugging a long run. + */ +export async function chat( + config: LlmConfig, + model: string, + opts: ChatOptions, +): Promise { + const headers: Record = { 'Content-Type': 'application/json' }; + // Ollama ignores Authorization; sending it only when present keeps the + // no-key local path clean and still satisfies hosted providers. + if (config.apiKey) headers['Authorization'] = `Bearer ${config.apiKey}`; + + const body: Record = { + model, + messages: [ + { role: 'system', content: opts.system }, + { role: 'user', content: opts.user }, + ], + temperature: opts.temperature, + }; + if (opts.maxTokens !== undefined) body['max_tokens'] = opts.maxTokens; + if (opts.json) body['response_format'] = { type: 'json_object' }; + + const response = await fetch(`${config.baseUrl}/chat/completions`, { + method: 'POST', + headers, + body: JSON.stringify(body), + signal: AbortSignal.timeout(config.timeoutMs), + }); + + if (!response.ok) { + const detail = await response.text().catch(() => ''); + throw new Error( + `LLM request failed (${response.status} ${response.statusText}) at ${config.baseUrl} for model "${model}": ${detail.slice(0, 300)}`, + ); + } + + const result = await response.json() as { + choices?: Array<{ message?: { content?: string } }>; + }; + const content = result.choices?.[0]?.message?.content; + if (typeof content !== 'string') { + throw new Error(`LLM returned no message content for model "${model}"`); + } + return content; +} diff --git a/benchmarks/lib/metrics.ts b/benchmarks/lib/metrics.ts index be44ed6..1857eb3 100644 --- a/benchmarks/lib/metrics.ts +++ b/benchmarks/lib/metrics.ts @@ -1,28 +1,72 @@ // Benchmark scoring utilities — Recall@k, latency stats, session extraction +// +// Two Recall@k definitions live here, and the distinction is the whole +// ballgame for comparability. +// +// LongMemEval's corpus unit is the *session*: retrieve k sessions, score a hit +// if a gold session is among them. MemForge's retrieval unit is the warm-tier +// *row*, and consolidation packs many sessions into one row — up to +// CONSOLIDATION_INNER_BATCH_SIZE of them. So "top 5 rows" can contain +// hundreds of sessions, and scoring that as R@5 measures something far more +// generous than the paper does. +// +// Both numbers are therefore computed and reported under distinct names: +// +// recallAtKSessions — first k distinct sessions in rank order. This is the +// paper-comparable definition and the headline metric. +// recallAtKRows — any gold session anywhere inside the top-k rows. This +// is MemForge's native retrieval behaviour and is +// strictly >= the session number; the gap is the packing +// advantage, quantified by sessionsPerRow. +// +// Run with CONSOLIDATION_INNER_BATCH_SIZE=1 to make rows and sessions 1:1, at +// which point the two metrics converge. import type { QuestionResult, LatencyStats, CategoryResult } from '../longmemeval/types.js'; const SESSION_ID_RE = /\[SESSION_ID:([^\]]+)\]/g; -/** Extract [SESSION_ID:xxx] markers from warm-tier content. */ +/** Extract [SESSION_ID:xxx] markers from warm-tier content, in order of appearance. */ export function extractSessionIds(content: string): string[] { - const ids: string[] = []; - let match; - while ((match = SESSION_ID_RE.exec(content)) !== null) { - if (match[1]) ids.push(match[1]); - } - SESSION_ID_RE.lastIndex = 0; - return ids; + // matchAll rather than exec-with-lastIndex-reset: the shared /g regex + // carries mutable state between calls, and an early return would leave + // lastIndex dangling for the next caller. + return [...content.matchAll(SESSION_ID_RE)] + .map((m) => m[1]) + .filter((id): id is string => id !== undefined); } /** - * Compute Recall@k: 1 if any answer session appears in the top-k retrieved sessions, 0 otherwise. - * retrievedSessionIds should be in rank order (from highest to lowest ranked results). + * Recall@k over sessions — 1 if any gold session is among the first k distinct + * sessions in rank order, else 0. This is LongMemEval's definition. + * + * `retrievedSessionIds` must be rank-ordered and already de-duplicated. */ -export function recallAtK(retrievedSessionIds: string[], answerSessionIds: string[], k: number): number { +export function recallAtKSessions( + retrievedSessionIds: string[], + answerSessionIds: string[], + k: number, +): number { const topK = new Set(retrievedSessionIds.slice(0, k)); - for (const aid of answerSessionIds) { - if (topK.has(String(aid))) return 1; + return answerSessionIds.some((aid) => topK.has(String(aid))) ? 1 : 0; +} + +/** + * Recall@k over rows — 1 if any gold session appears anywhere inside the top-k + * retrieved rows, else 0. Generous relative to the session metric whenever + * consolidation packs multiple sessions per row; reported so the packing + * advantage is visible rather than baked silently into a headline number. + * + * `perRowSessionIds[i]` holds the sessions found in the i-th ranked row. + */ +export function recallAtKRows( + perRowSessionIds: string[][], + answerSessionIds: string[], + k: number, +): number { + const gold = new Set(answerSessionIds.map(String)); + for (const row of perRowSessionIds.slice(0, k)) { + if (row.some((sid) => gold.has(sid))) return 1; } return 0; } @@ -55,12 +99,25 @@ export function latencyStats(values: number[]): LatencyStats { export function aggregateScores( results: QuestionResult[], topKValues: number[], -): { overall: { recallAt: Record; queryLatency: LatencyStats; ingestLatency: LatencyStats }; perCategory: Record } { +): { + overall: { + recallAtSessions: Record; + recallAtRows: Record; + sessionsPerRow: number; + queryLatency: LatencyStats; + ingestLatency: LatencyStats; + }; + perCategory: Record; +} { + const mean = (xs: number[]): number => + xs.length > 0 ? xs.reduce((a, b) => a + b, 0) / xs.length : 0; + // Overall - const overallRecall: Record = {}; + const overallSessions: Record = {}; + const overallRows: Record = {}; for (const k of topKValues) { - const recalls = results.map((r) => r.recallAt[k] ?? 0); - overallRecall[k] = recalls.length > 0 ? recalls.reduce((a, b) => a + b, 0) / recalls.length : 0; + overallSessions[k] = mean(results.map((r) => r.recallAtSessions[k] ?? 0)); + overallRows[k] = mean(results.map((r) => r.recallAtRows[k] ?? 0)); } const queryLatencies = results.map((r) => r.latency.queryMs); @@ -77,21 +134,25 @@ export function aggregateScores( const perCategory: Record = {}; for (const [cat, catResults] of categories) { - const catRecall: Record = {}; + const catSessions: Record = {}; + const catRows: Record = {}; for (const k of topKValues) { - const recalls = catResults.map((r) => r.recallAt[k] ?? 0); - catRecall[k] = recalls.reduce((a, b) => a + b, 0) / recalls.length; + catSessions[k] = mean(catResults.map((r) => r.recallAtSessions[k] ?? 0)); + catRows[k] = mean(catResults.map((r) => r.recallAtRows[k] ?? 0)); } perCategory[cat] = { count: catResults.length, - recallAt: catRecall, + recallAtSessions: catSessions, + recallAtRows: catRows, latency: latencyStats(catResults.map((r) => r.latency.queryMs)), }; } return { overall: { - recallAt: overallRecall, + recallAtSessions: overallSessions, + recallAtRows: overallRows, + sessionsPerRow: mean(results.map((r) => r.sessionsPerRow)), queryLatency: latencyStats(queryLatencies), ingestLatency: latencyStats(ingestLatencies), }, diff --git a/benchmarks/longmemeval-qa/README.md b/benchmarks/longmemeval-qa/README.md index 92286bb..6f15b66 100644 --- a/benchmarks/longmemeval-qa/README.md +++ b/benchmarks/longmemeval-qa/README.md @@ -3,8 +3,13 @@ This benchmark runs the **full LongMemEval pipeline**: retrieve → generate answer → LLM judge. **Official metric:** End-to-end QA accuracy (not retrieval R@5) -**Judge:** `gpt-4o-2024-08-06` (per paper's protocol, >97% human agreement) -**Reader:** `gpt-4o-2024-08-06` (configurable via `QA_READER_MODEL`) +**Judge (default):** `qwen3.5:cloud` via local Ollama — free, no API key +**Reader (default):** `qwen3.5:cloud` via local Ollama + +> Only a `gpt-4o*` judge follows the paper's protocol (>97% human agreement). +> With any other judge these scores track relative progress but are **not** +> comparable to published LongMemEval numbers. The runner warns, the report +> says so, and each manifest records the judge used. ## Why This Matters @@ -15,16 +20,18 @@ The official LongMemEval metric is **QA accuracy**, not retrieval Recall@5. Retr ## Quick Run (10 questions) ```bash -OPENAI_API_KEY=sk-... BENCHMARK_LIMIT=10 npm run benchmark:longmemeval-qa +BENCHMARK_LIMIT=10 npm run benchmark:longmemeval-qa ``` ## Full Run (500 questions) ```bash -OPENAI_API_KEY=sk-... npm run benchmark:longmemeval-qa +npm run benchmark:longmemeval-qa ``` -> **Warning:** A full 500-question run costs ~$50–100 in OpenAI API charges (judge + reader calls). +> **Cost:** $0 on the default Ollama path. A full 500-question run against +> OpenAI (`QA_API_BASE=https://api.openai.com/v1`, `QA_JUDGE_MODEL=gpt-4o-2024-08-06`) +> costs ~$50–100 in judge + reader calls. ## Configuration @@ -34,9 +41,12 @@ All via environment variables: |----------|---------|-------------| | `MEMFORGE_URL` | `http://localhost:3333` | MemForge server URL | | `MEMFORGE_TOKEN` | (none) | Bearer token for auth | -| `OPENAI_API_KEY` | (required) | OpenAI API key for judge and reader | -| `QA_JUDGE_MODEL` | `gpt-4o-2024-08-06` | Judge model (must be >= GPT-4o quality) | -| `QA_READER_MODEL` | `gpt-4o-2024-08-06` | Reader/generator model | +| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama host root (no `/v1`) | +| `QA_API_BASE` | `$OLLAMA_BASE_URL/v1` | OpenAI-compatible endpoint | +| `QA_API_KEY` | `$OPENAI_API_KEY` | Bearer token; unnecessary for Ollama | +| `QA_TIMEOUT_MS` | `180000` | Per-request timeout | +| `QA_JUDGE_MODEL` | `qwen3.5:cloud` | Judge model (use `gpt-4o-*` for comparable results) | +| `QA_READER_MODEL` | `qwen3.5:cloud` | Reader/generator model | | `BENCHMARK_LIMIT` | `500` | Number of questions to evaluate | | `BENCHMARK_OFFSET` | `0` | Skip first N questions | | `BENCHMARK_MODES` | `hybrid` | Retrieval mode: keyword, semantic, hybrid | diff --git a/benchmarks/longmemeval-qa/evaluate.ts b/benchmarks/longmemeval-qa/evaluate.ts index 5692898..ba01fc1 100644 --- a/benchmarks/longmemeval-qa/evaluate.ts +++ b/benchmarks/longmemeval-qa/evaluate.ts @@ -2,18 +2,24 @@ // // Runs the full LongMemEval pipeline: retrieve → generate answer → LLM judge // Official metric: end-to-end QA accuracy (not retrieval R@5) -// Judge: gpt-4o-2024-08-06 (per paper's protocol, >97% human agreement) // -// Usage: npx tsx benchmarks/longmemeval-qa/evaluate.ts [--limit=100] [--judge=openai] +// Judge and reader default to Ollama (qwen3.5:cloud) so a run costs nothing +// and needs no API key. Scores from a non-GPT-4o judge track relative +// progress but are NOT comparable to published LongMemEval numbers — see +// ../lib/llm.ts and the banner emitted below. +// +// Usage: npx tsx benchmarks/longmemeval-qa/evaluate.ts [--limit=100] import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs'; import { join } from 'node:path'; import { performance } from 'node:perf_hooks'; import { loadConfig, type BenchmarkConfig } from '../lib/config.js'; +import { chat, loadLlmConfig, isPaperProtocolJudge } from '../lib/llm.js'; +import type { QueryMode } from '../../src/types.js'; import type { LongMemEvalInstance, QAQuestionResult } from './types.js'; -const JUDGE_MODEL = process.env['QA_JUDGE_MODEL'] ?? 'gpt-4o-2024-08-06'; -const OPENAI_API_KEY = process.env['OPENAI_API_KEY']; +const LLM = loadLlmConfig(); +const JUDGE_MODEL = LLM.judgeModel; async function createClient(config: BenchmarkConfig) { const { MemForgeClient } = await import('../../src/client.js'); @@ -28,50 +34,37 @@ async function judgeAnswer( expectedAnswer: string, generatedAnswer: string, ): Promise<{ correct: boolean; score: number; reasoning: string }> { - if (!OPENAI_API_KEY) { - throw new Error('OPENAI_API_KEY required for QA judging. Set QA_JUDGE_MODEL to use alternative judge.'); - } - - // Use OpenAI API directly to avoid external SDK dependency - const response = await fetch('https://api.openai.com/v1/chat/completions', { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - 'Authorization': `Bearer ${OPENAI_API_KEY}`, - }, - body: JSON.stringify({ - model: JUDGE_MODEL, - messages: [ - { - role: 'system', - content: `You are an expert judge evaluating AI agent answers against gold standard answers. + const content = await chat(LLM, JUDGE_MODEL, { + system: `You are an expert judge evaluating AI agent answers against gold standard answers. Rate the generated answer on a scale of 0.0 to 1.0 based on factual correctness and completeness. Score 1.0 = fully correct and complete, 0.0 = completely wrong or irrelevant. Consider partial credit for answers that contain some correct information. Output JSON: {"score": 0.0-1.0, "correct": true/false, "reasoning": "brief explanation"}`, - }, - { - role: 'user', - content: `Question: ${question}\n\nGold Answer: ${expectedAnswer}\n\nGenerated Answer: ${generatedAnswer}`, - }, - ], - temperature: 0.0, - response_format: { type: 'json_object' }, - }), + user: `Question: ${question}\n\nGold Answer: ${expectedAnswer}\n\nGenerated Answer: ${generatedAnswer}`, + temperature: 0.0, + json: true, }); - if (!response.ok) { - throw new Error(`Judge API error: ${response.status} ${response.statusText}`); + let parsed: { score?: unknown; correct?: unknown; reasoning?: unknown }; + try { + parsed = JSON.parse(content); + } catch { + // Smaller judges occasionally wrap JSON in prose despite json mode. Fail + // loudly with the payload rather than scoring the question 0 — a silent + // zero is indistinguishable from a genuinely wrong answer and would + // depress the headline accuracy for a transport-level reason. + throw new Error(`Judge "${JUDGE_MODEL}" returned non-JSON: ${content.slice(0, 200)}`); } - const result = await response.json(); - const content = result.choices[0].message.content; - const parsed = JSON.parse(content); + const score = typeof parsed.score === 'number' ? parsed.score : Number(parsed.score); + if (!Number.isFinite(score)) { + throw new Error(`Judge "${JUDGE_MODEL}" returned no usable score: ${content.slice(0, 200)}`); + } return { - score: parsed.score, - correct: parsed.correct, - reasoning: parsed.reasoning, + score, + correct: typeof parsed.correct === 'boolean' ? parsed.correct : score >= 0.5, + reasoning: typeof parsed.reasoning === 'string' ? parsed.reasoning : '', }; } @@ -80,47 +73,18 @@ async function generateAnswer( retrievedContext: string[], readerModel?: string, ): Promise { - const model = readerModel ?? process.env['QA_READER_MODEL'] ?? 'gpt-4o-2024-08-06'; - const apiKey = process.env['OPENAI_API_KEY']; - - if (!apiKey) { - throw new Error('OPENAI_API_KEY required for answer generation'); - } - + const model = readerModel ?? LLM.readerModel; const context = retrievedContext.join('\n\n---\n\n'); - const response = await fetch('https://api.openai.com/v1/chat/completions', { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - 'Authorization': `Bearer ${apiKey}`, - }, - body: JSON.stringify({ - model: model, - messages: [ - { - role: 'system', - content: `You are an AI agent answering questions based on retrieved context. + return chat(LLM, model, { + system: `You are an AI agent answering questions based on retrieved context. Use ONLY the information in the provided context to answer. If the context doesn't contain enough information, say so clearly. Be concise and factual. Cite specific details from the context.`, - }, - { - role: 'user', - content: `Context:\n${context}\n\nQuestion: ${question}\n\nAnswer:`, - }, - ], - temperature: 0.3, - max_tokens: 500, - }), + user: `Context:\n${context}\n\nQuestion: ${question}\n\nAnswer:`, + temperature: 0.3, + maxTokens: 500, }); - - if (!response.ok) { - throw new Error(`Reader API error: ${response.status} ${response.statusText}`); - } - - const result = await response.json(); - return result.choices[0].message.content ?? ''; } async function evaluateQuestion( @@ -137,7 +101,7 @@ async function evaluateQuestion( const results = await client.query(agentId, { q: instance.question, limit: maxK, - mode: config.queryModes[0] ?? 'hybrid', + mode: (config.queryModes[0] ?? 'hybrid') as QueryMode, }); const queryMs = performance.now() - queryStart; @@ -211,15 +175,22 @@ export async function main(configOverride?: BenchmarkConfig): Promise = []; + // Accumulates across every instance — the manifest below reports the full + // set. Previously declared inside the loop and read after it, which is a + // ReferenceError at runtime. + const allSessions = new Set(); for (let i = 0; i < instances.length; i++) { const instance = instances[i]; - const allSessions = new Set(); + if (!instance) continue; try { // Ingest all haystack sessions for this question diff --git a/benchmarks/longmemeval-qa/run.ts b/benchmarks/longmemeval-qa/run.ts index 3384d46..6e89790 100644 --- a/benchmarks/longmemeval-qa/run.ts +++ b/benchmarks/longmemeval-qa/run.ts @@ -8,6 +8,9 @@ import { join } from 'node:path'; import { loadConfig, type BenchmarkConfig } from '../lib/config.js'; import { main as evaluate } from './evaluate.js'; import type { QAQuestionResult, QAAggregateResult } from './types.js'; +import { loadLlmConfig, isPaperProtocolJudge } from '../lib/llm.js'; + +const LLM = loadLlmConfig(); function calculateStats(results: QAQuestionResult[]): QAAggregateResult { const totalQuestions = results.length; @@ -18,12 +21,11 @@ function calculateStats(results: QAQuestionResult[]): QAAggregateResult { // Per-category breakdown const perCategory: Record = {}; for (const result of results) { - if (!perCategory[result.questionType]) { - perCategory[result.questionType] = { count: 0, correct: 0, accuracy: 0, avgScore: 0 }; - } - perCategory[result.questionType].count++; - if (result.correct) perCategory[result.questionType].correct++; - perCategory[result.questionType].avgScore += result.score ?? 0; + const bucket = perCategory[result.questionType] + ?? (perCategory[result.questionType] = { count: 0, correct: 0, accuracy: 0, avgScore: 0 }); + bucket.count++; + if (result.correct) bucket.correct++; + bucket.avgScore += result.score ?? 0; } for (const category of Object.values(perCategory)) { @@ -50,8 +52,8 @@ function calculateStats(results: QAQuestionResult[]): QAAggregateResult { perCategory, latency: avgLatency, tokensPerRetrieval, - judgeModel: process.env['QA_JUDGE_MODEL'] ?? 'gpt-4o-2024-08-06', - readerModel: process.env['QA_READER_MODEL'] ?? 'gpt-4o-2024-08-06', + judgeModel: LLM.judgeModel, + readerModel: LLM.readerModel, timestamp: new Date().toISOString(), }; } @@ -63,9 +65,21 @@ function generateReport(stats: QAAggregateResult, results: QAQuestionResult[]): lines.push(''); lines.push(`Generated: ${stats.timestamp}`); lines.push(''); - lines.push('> **Important:** These are **end-to-end QA accuracy** scores (retrieve → generate → judge).'); - lines.push('> This is the official LongMemEval metric, directly comparable to leaderboard entries.'); - lines.push('> Judge: `' + stats.judgeModel + '` (per paper protocol, >97% human agreement)'); + lines.push('> **Important:** These are **end-to-end QA accuracy** scores (retrieve → generate → judge),'); + lines.push('> not retrieval recall.'); + lines.push(''); + if (isPaperProtocolJudge(stats.judgeModel)) { + lines.push('> Judge: `' + stats.judgeModel + '` — the paper protocol judge (>97% human agreement),'); + lines.push('> so these scores are directly comparable to published LongMemEval entries.'); + } else { + // The comparability claim is conditional on the judge. Emitting it + // unconditionally is how a local-model score gets quoted as a + // leaderboard result. + lines.push('> **Not comparable to published LongMemEval numbers.** Judge: `' + stats.judgeModel + '`,'); + lines.push('> whereas the paper protocol judges with `gpt-4o` (>97% human agreement). These'); + lines.push('> scores are valid for tracking relative progress between MemForge runs only.'); + lines.push('> Re-run with `QA_JUDGE_MODEL=gpt-4o-2024-08-06` before publishing or comparing.'); + } lines.push(''); lines.push('## Overall Accuracy'); lines.push(''); @@ -123,7 +137,7 @@ export async function main() { // Parse CLI args for (const arg of process.argv.slice(2)) { const limitMatch = arg.match(/^--limit=(\d+)$/); - if (limitMatch?.[1]) config.limit = parseInt(limitMatch[1], 10); + if (limitMatch?.[1]) config.questionLimit = parseInt(limitMatch[1], 10); } console.log('=== LongMemEval QA Accuracy Benchmark ==='); diff --git a/benchmarks/longmemeval/evaluate.ts b/benchmarks/longmemeval/evaluate.ts index b668b77..d760795 100644 --- a/benchmarks/longmemeval/evaluate.ts +++ b/benchmarks/longmemeval/evaluate.ts @@ -7,7 +7,7 @@ import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs'; import { join } from 'node:path'; import { performance } from 'node:perf_hooks'; import { loadConfig, type BenchmarkConfig } from '../lib/config.js'; -import { extractSessionIds, recallAtK } from '../lib/metrics.js'; +import { extractSessionIds, recallAtKSessions, recallAtKRows } from '../lib/metrics.js'; import type { LongMemEvalInstance, QuestionResult, IngestManifest } from './types.js'; async function createClient(config: BenchmarkConfig) { @@ -38,40 +38,39 @@ async function evaluateQuestion( }); const queryMs = performance.now() - queryStart; - // Extract session IDs from results in rank order - const retrievedSessionIds: string[] = []; - for (const result of results) { + // Sessions per retrieved row, in rank order. Both Recall@k definitions and + // the packing factor derive from this one structure. + const perRowSessionIds: string[][] = results.map((result) => { const content = typeof result === 'object' && result !== null && 'content' in result ? (result as { content: string }).content : ''; - const extracted = extractSessionIds(content); - for (const id of extracted) { - if (!retrievedSessionIds.includes(id)) { + return extractSessionIds(content); + }); + + // Flatten to a rank-ordered, de-duplicated session list. + const seen = new Set(); + const retrievedSessionIds: string[] = []; + for (const row of perRowSessionIds) { + for (const id of row) { + if (!seen.has(id)) { + seen.add(id); retrievedSessionIds.push(id); } } } - // Compute Recall@k for each k - const recallAt: Record = {}; const answerIds = instance.answer_session_ids.map(String); + const recallAtSessions: Record = {}; + const recallAtRows: Record = {}; for (const k of config.queryTopK) { - // For Recall@k, use session IDs from the top-k warm-tier results - // Since one warm-tier result may contain multiple sessions (batch consolidation), - // we extract all session IDs from results[0..k-1] - const topKSessionIds: string[] = []; - for (let i = 0; i < Math.min(k, results.length); i++) { - const r = results[i]; - const content = typeof r === 'object' && r !== null && 'content' in r - ? (r as { content: string }).content - : ''; - for (const sid of extractSessionIds(content)) { - if (!topKSessionIds.includes(sid)) topKSessionIds.push(sid); - } - } - recallAt[k] = recallAtK(topKSessionIds, answerIds, topKSessionIds.length); + recallAtSessions[k] = recallAtKSessions(retrievedSessionIds, answerIds, k); + recallAtRows[k] = recallAtKRows(perRowSessionIds, answerIds, k); } + const sessionsPerRow = perRowSessionIds.length > 0 + ? perRowSessionIds.reduce((sum, row) => sum + row.length, 0) / perRowSessionIds.length + : 0; + return { questionIndex, questionType: instance.question_type ?? 'unknown', @@ -79,7 +78,9 @@ async function evaluateQuestion( expectedAnswer: instance.answer, answerSessionIds: instance.answer_session_ids, retrievedSessionIds, - recallAt, + recallAtSessions, + recallAtRows, + sessionsPerRow, latency: { ingestMs, consolidateMs, queryMs }, queryMode: mode, resultCount: results.length, diff --git a/benchmarks/longmemeval/report.ts b/benchmarks/longmemeval/report.ts index ac12b13..cb854e6 100644 --- a/benchmarks/longmemeval/report.ts +++ b/benchmarks/longmemeval/report.ts @@ -41,16 +41,28 @@ function generateMarkdown(reports: BenchmarkReport[]): string { // Overall Recall table lines.push('### Retrieval Quality'); lines.push(''); - lines.push('| Metric | Score |'); - lines.push('|--------|-------|'); - const ks = Object.keys(report.overall.recallAt).map(Number).sort((a, b) => a - b); + lines.push('> **Recall@k (sessions)** is the comparable metric: a hit means a gold'); + lines.push('> session is among the first k distinct sessions by rank — LongMemEval\'s'); + lines.push('> definition. **Recall@k (rows)** counts a hit anywhere inside the top-k'); + lines.push('> retrieved rows; because consolidation packs multiple sessions per row it'); + lines.push('> is strictly more generous and is NOT comparable to published figures.'); + lines.push(''); + lines.push('| Metric | Sessions (comparable) | Rows (native) |'); + lines.push('|--------|----------------------|---------------|'); + const ks = Object.keys(report.overall.recallAtSessions).map(Number).sort((a, b) => a - b); for (const k of ks) { - lines.push(`| Recall@${k} | ${formatPct(report.overall.recallAt[k] ?? 0)} |`); + const sess = formatPct(report.overall.recallAtSessions[k] ?? 0); + const rows = formatPct(report.overall.recallAtRows[k] ?? 0); + lines.push(`| Recall@${k} | ${sess} | ${rows} |`); } lines.push(''); + lines.push(`Sessions packed per retrieved row: **${report.overall.sessionsPerRow.toFixed(1)}**`); + lines.push('(1.0 means rows and sessions are 1:1 and the two columns converge.)'); + lines.push(''); - // Baseline comparison - lines.push('**Baselines:** Hippo 74.0% R@5 (BM25 keyword), Zep +18.5% over full-context'); + // Baseline comparison — only the sessions column is on the same footing. + lines.push('**Baselines (compare against the Sessions column only):** Hippo 74.0% R@5'); + lines.push('(BM25 keyword), Zep +18.5% over full-context'); lines.push(''); // Per-category table @@ -58,11 +70,11 @@ function generateMarkdown(reports: BenchmarkReport[]): string { lines.push(''); const categories = Object.entries(report.perCategory).sort(([a], [b]) => a.localeCompare(b)); if (categories.length > 0) { - const headerKs = ks.map((k) => `R@${k}`).join(' | '); + const headerKs = ks.map((k) => `R@${k} (sessions)`).join(' | '); lines.push(`| Category | Count | ${headerKs} |`); lines.push(`|----------|-------|${ks.map(() => '------').join('|')}|`); for (const [cat, data] of categories) { - const recalls = ks.map((k) => formatPct(data.recallAt[k] ?? 0)).join(' | '); + const recalls = ks.map((k) => formatPct(data.recallAtSessions[k] ?? 0)).join(' | '); lines.push(`| ${cat} | ${data.count} | ${recalls} |`); } } diff --git a/benchmarks/longmemeval/types.ts b/benchmarks/longmemeval/types.ts index 464ea9a..7ab0f68 100644 --- a/benchmarks/longmemeval/types.ts +++ b/benchmarks/longmemeval/types.ts @@ -25,7 +25,12 @@ export interface QuestionResult { expectedAnswer: string; answerSessionIds: number[]; retrievedSessionIds: string[]; - recallAt: Record; + /** Paper-comparable: gold session among the first k distinct sessions by rank. */ + recallAtSessions: Record; + /** MemForge-native: gold session anywhere inside the top-k retrieved rows. */ + recallAtRows: Record; + /** Distinct sessions packed into each retrieved row — explains the gap above. */ + sessionsPerRow: number; latency: { ingestMs: number; consolidateMs: number; @@ -46,7 +51,8 @@ export interface LatencyStats { export interface CategoryResult { count: number; - recallAt: Record; + recallAtSessions: Record; + recallAtRows: Record; latency: LatencyStats; } @@ -57,7 +63,9 @@ export interface BenchmarkReport { queryMode: string; consolidationMode: string; overall: { - recallAt: Record; + recallAtSessions: Record; + recallAtRows: Record; + sessionsPerRow: number; queryLatency: LatencyStats; ingestLatency: LatencyStats; }; diff --git a/funding.json b/funding.json index 8dd8777..223337e 100644 --- a/funding.json +++ b/funding.json @@ -1,128 +1,167 @@ { - "$schema": "https://fundingjson.org/schema/v1.1.0/funding.schema.json", - "entity": { - "type": "individual", - "role": "maintainer", - "name": "John Brooke", - "email": "john@salishforge.com", - "description": "Solo maintainer building an open-source, MIT-licensed stack for long-running, multi-agent AI systems. The thesis is that frontier model labs are rightly focused on raw capability, but LLMs need a substrate that accumulates experience and revises beliefs over time to operate on human timelines. I'm building that substrate — security-first, zero-trust from day one — out of pocket on a modest VPS, with two autonomous agents running in production on the stack. Shipping weekly. Based in Seattle, WA.", - "webpageUrl": { - "url": "https://github.com/salishforge" - } + "$schema": "https://fundingjson.org/schema/v1.1.0/funding.schema.json", + "entity": { + "type": "individual", + "role": "maintainer", + "name": "John Brooke", + "email": "john@salishforge.com", + "description": "Solo maintainer building an open-source, MIT-licensed stack for long-running, multi-agent AI systems. The thesis is that frontier model labs are rightly focused on raw capability, but LLMs need a substrate that accumulates experience and revises beliefs over time to operate on human timelines. I'm building that substrate \u2014 security-first, zero-trust from day one \u2014 out of pocket on a modest VPS, with two autonomous agents running in production on the stack. Shipping weekly. Based in Seattle, WA.", + "webpageUrl": { + "url": "https://github.com/salishforge" + } + }, + "projects": [ + { + "guid": "memforge", + "name": "MemForge", + "description": "Neuroscience-inspired memory system for AI agents. Unlike passive vector stores, MemForge treats memory quality as something that should actively improve over time. It runs a 10-phase 'sleep cycle' during idle periods: scoring, triage, conflict resolution, LLM-driven revision of low-confidence entries, graph maintenance, reflection, schema crystallization, meta-reflection, and gap analysis.\n\nBenchmarked against LongMemEval (ICLR 2025) with p50 32ms / p95 47ms hybrid-search latency; retrieval accuracy figures are being re-measured after a scorer correction. Ships as a Docker standalone image, Python SDK, TypeScript SDK, and a 17-tool MCP server with integrations for Claude Desktop, Microsoft 365 Copilot, ChatGPT, LangChain, and CrewAI. Cleared 9 rounds of adversarial security review at MEDIUM+, with a published threat model, RLS, prompt-injection boundaries, SSRF prevention, and HMAC-chained audit logs.\n\nBuilt on PostgreSQL + pgvector (halfvec float16) with local embeddings via Transformers.js and Ollama support for fully self-hosted deployments.", + "webpageUrl": { + "url": "https://github.com/salishforge/memforge" + }, + "repositoryUrl": { + "url": "https://github.com/salishforge/memforge" + }, + "licenses": [ + "spdx:MIT" + ], + "tags": [ + "ai", + "agents", + "memory", + "agent-memory", + "mcp", + "postgresql", + "pgvector", + "typescript", + "knowledge-graph", + "infrastructure" + ] }, - "projects": [ - { - "guid": "memforge", - "name": "MemForge", - "description": "Neuroscience-inspired memory system for AI agents. Unlike passive vector stores, MemForge treats memory quality as something that should actively improve over time. It runs a 10-phase 'sleep cycle' during idle periods: scoring, triage, conflict resolution, LLM-driven revision of low-confidence entries, graph maintenance, reflection, schema crystallization, meta-reflection, and gap analysis.\n\nScores 93.2% Recall@5 on LongMemEval (ICLR 2025) with p50 32ms / p95 47ms hybrid-search latency. Ships as a Docker standalone image, Python SDK, TypeScript SDK, and a 17-tool MCP server with integrations for Claude Desktop, Microsoft 365 Copilot, ChatGPT, LangChain, and CrewAI. Cleared 9 rounds of adversarial security review at MEDIUM+, with a published threat model, RLS, prompt-injection boundaries, SSRF prevention, and HMAC-chained audit logs.\n\nBuilt on PostgreSQL + pgvector (halfvec float16) with local embeddings via Transformers.js and Ollama support for fully self-hosted deployments.", - "webpageUrl": { - "url": "https://github.com/salishforge/memforge" - }, - "repositoryUrl": { - "url": "https://github.com/salishforge/memforge" - }, - "licenses": ["spdx:MIT"], - "tags": ["ai", "agents", "memory", "agent-memory", "mcp", "postgresql", "pgvector", "typescript", "knowledge-graph", "infrastructure"] - }, - { - "guid": "engram", - "name": "Engram", - "description": "Agent Memory Intelligence Benchmark. Existing benchmarks reward flashy retrieval metrics (Recall@k) that don't correlate with downstream task quality. Engram measures what actually matters: does the agent perform its job better with this memory system than without it?\n\nA three-tier evaluation framework weighted 20/40/40: retrieval quality, knowledge management (temporal accuracy, contradiction resolution, long-horizon retention, staleness detection, context efficiency), and actual agent task performance delta. Adapter-based architecture so any memory system can participate by implementing four methods. Positioned to fill gaps in existing benchmarks including LongMemEval, LoCoMo, MemoryAgentBench, and MEMTRACK.\n\nMaintained deliberately alongside MemForge so my own system — and competitors — can be evaluated honestly and publicly.", - "webpageUrl": { - "url": "https://github.com/salishforge/engram" - }, - "repositoryUrl": { - "url": "https://github.com/salishforge/engram" - }, - "licenses": ["spdx:MIT"], - "tags": ["ai", "agents", "benchmark", "evaluation", "agent-memory", "memory", "llm", "research", "infrastructure"] - } + { + "guid": "engram", + "name": "Engram", + "description": "Agent Memory Intelligence Benchmark. Existing benchmarks reward flashy retrieval metrics (Recall@k) that don't correlate with downstream task quality. Engram measures what actually matters: does the agent perform its job better with this memory system than without it?\n\nA three-tier evaluation framework weighted 20/40/40: retrieval quality, knowledge management (temporal accuracy, contradiction resolution, long-horizon retention, staleness detection, context efficiency), and actual agent task performance delta. Adapter-based architecture so any memory system can participate by implementing four methods. Positioned to fill gaps in existing benchmarks including LongMemEval, LoCoMo, MemoryAgentBench, and MEMTRACK.\n\nMaintained deliberately alongside MemForge so my own system \u2014 and competitors \u2014 can be evaluated honestly and publicly.", + "webpageUrl": { + "url": "https://github.com/salishforge/engram" + }, + "repositoryUrl": { + "url": "https://github.com/salishforge/engram" + }, + "licenses": [ + "spdx:MIT" + ], + "tags": [ + "ai", + "agents", + "benchmark", + "evaluation", + "agent-memory", + "memory", + "llm", + "research", + "infrastructure" + ] + } + ], + "funding": { + "channels": [ + { + "guid": "direct-email", + "type": "other", + "address": "mailto:john@salishforge.com", + "description": "For grants, compute credits, corporate sponsorships, or larger direct transfers. Reach out via email and I'll route to the appropriate payment method." + }, + { + "guid": "github-sponsors", + "type": "payment-provider", + "address": "https://github.com/sponsors/salishforge", + "description": "Recurring and one-time sponsorships via GitHub Sponsors. Supports all tiered plans below." + } ], - "funding": { + "plans": [ + { + "guid": "goodwill", + "status": "active", + "name": "Goodwill", + "description": "Any amount, one-time. Every dollar offsets inference costs so development doesn't pause for weekly rate limits to reset.", + "amount": 0, + "currency": "USD", + "frequency": "one-time", "channels": [ - { - "guid": "direct-email", - "type": "other", - "address": "mailto:john@salishforge.com", - "description": "For grants, compute credits, corporate sponsorships, or larger direct transfers. Reach out via email and I'll route to the appropriate payment method." - }, - { - "guid": "github-sponsors", - "type": "payment-provider", - "address": "https://github.com/sponsors/salishforge", - "description": "Recurring and one-time sponsorships via GitHub Sponsors. Supports all tiered plans below." - } - ], - "plans": [ - { - "guid": "goodwill", - "status": "active", - "name": "Goodwill", - "description": "Any amount, one-time. Every dollar offsets inference costs so development doesn't pause for weekly rate limits to reset.", - "amount": 0, - "currency": "USD", - "frequency": "one-time", - "channels": ["github-sponsors", "direct-email"] - }, - { - "guid": "supporter", - "status": "active", - "name": "Supporter", - "description": "Helps cover monthly inference costs that would otherwise throttle development.", - "amount": 5, - "currency": "USD", - "frequency": "monthly", - "channels": ["github-sponsors"] - }, - { - "guid": "builder", - "status": "active", - "name": "Builder", - "description": "Name or handle listed in project READMEs. Early access to benchmark write-ups and design notes.", - "amount": 25, - "currency": "USD", - "frequency": "monthly", - "channels": ["github-sponsors"] - }, - { - "guid": "sustainer", - "status": "active", - "name": "Sustainer", - "description": "Everything in Builder, plus a direct channel for integration questions about MemForge, Engram, or Hyphae.", - "amount": 100, - "currency": "USD", - "frequency": "monthly", - "channels": ["github-sponsors"] - }, - { - "guid": "company-sponsor", - "status": "active", - "name": "Company Sponsor", - "description": "Company logo in project READMEs and release notes. Priority integration support. Quarterly 30-minute call to discuss roadmap and your needs.", - "amount": 500, - "currency": "USD", - "frequency": "monthly", - "channels": ["github-sponsors", "direct-email"] - }, - { - "guid": "annual-infrastructure", - "status": "active", - "name": "Annual Infrastructure Grant", - "description": "Covers one full year of infrastructure and inference costs: Claude Max subscription, open-weight GPU compute for benchmark runs, targeted proprietary API runs for comparative data, and VPS hosting. Ideal for grant programs or corporate sponsors looking to fund a full development year.", - "amount": 15000, - "currency": "USD", - "frequency": "yearly", - "channels": ["direct-email"] - } - ], - "history": [ - { - "year": 2026, - "income": 0, - "expenses": 500, - "currency": "USD", - "description": "Fully self-funded since project inception in March 2026. Approximately $500 in out-of-pocket LLM inference and infrastructure costs to date, with deliberate development pauses to let weekly rate limits reset. Does not yet include compute costs for intensive benchmarking, load testing, or model training." - } + "github-sponsors", + "direct-email" ] - } + }, + { + "guid": "supporter", + "status": "active", + "name": "Supporter", + "description": "Helps cover monthly inference costs that would otherwise throttle development.", + "amount": 5, + "currency": "USD", + "frequency": "monthly", + "channels": [ + "github-sponsors" + ] + }, + { + "guid": "builder", + "status": "active", + "name": "Builder", + "description": "Name or handle listed in project READMEs. Early access to benchmark write-ups and design notes.", + "amount": 25, + "currency": "USD", + "frequency": "monthly", + "channels": [ + "github-sponsors" + ] + }, + { + "guid": "sustainer", + "status": "active", + "name": "Sustainer", + "description": "Everything in Builder, plus a direct channel for integration questions about MemForge, Engram, or Hyphae.", + "amount": 100, + "currency": "USD", + "frequency": "monthly", + "channels": [ + "github-sponsors" + ] + }, + { + "guid": "company-sponsor", + "status": "active", + "name": "Company Sponsor", + "description": "Company logo in project READMEs and release notes. Priority integration support. Quarterly 30-minute call to discuss roadmap and your needs.", + "amount": 500, + "currency": "USD", + "frequency": "monthly", + "channels": [ + "github-sponsors", + "direct-email" + ] + }, + { + "guid": "annual-infrastructure", + "status": "active", + "name": "Annual Infrastructure Grant", + "description": "Covers one full year of infrastructure and inference costs: Claude Max subscription, open-weight GPU compute for benchmark runs, targeted proprietary API runs for comparative data, and VPS hosting. Ideal for grant programs or corporate sponsors looking to fund a full development year.", + "amount": 15000, + "currency": "USD", + "frequency": "yearly", + "channels": [ + "direct-email" + ] + } + ], + "history": [ + { + "year": 2026, + "income": 0, + "expenses": 500, + "currency": "USD", + "description": "Fully self-funded since project inception in March 2026. Approximately $500 in out-of-pocket LLM inference and infrastructure costs to date, with deliberate development pauses to let weekly rate limits reset. Does not yet include compute costs for intensive benchmarking, load testing, or model training." + } + ] + } } diff --git a/package.json b/package.json index 5a05340..3ae2503 100644 --- a/package.json +++ b/package.json @@ -45,7 +45,7 @@ "test:dreams-compat": "node --import tsx/esm --test tests/dreams-compat.test.ts", "test:dreams-anthropic": "node --import tsx/esm --test tests/dreams-anthropic.test.ts", "test:dreams-bridge": "node --import tsx/esm --test tests/dreams-bridge.test.ts", - "test": "node --import tsx/esm --test --test-concurrency=1 tests/integration.test.ts tests/llm-paths.test.ts tests/http-api.test.ts tests/cache.test.ts tests/embedding-migration.test.ts tests/outcome-revision.test.ts tests/reflection-revision.test.ts tests/selective-forgetting.test.ts tests/multi-device.test.ts tests/dream-runs.test.ts tests/dreams-compat.test.ts tests/dreams-anthropic.test.ts tests/dreams-bridge.test.ts tests/sentiment-tagging.test.ts tests/adaptive-sleep.test.ts tests/epistemic-confidence.test.ts tests/explainable-memory.test.ts tests/causal-graph.test.ts tests/abstractions.test.ts tests/bootstrap.test.ts tests/contested-conflicts.test.ts", + "test": "node --import tsx/esm --test --test-concurrency=1 tests/integration.test.ts tests/llm-paths.test.ts tests/http-api.test.ts tests/cache.test.ts tests/embedding-migration.test.ts tests/outcome-revision.test.ts tests/reflection-revision.test.ts tests/selective-forgetting.test.ts tests/multi-device.test.ts tests/dream-runs.test.ts tests/dreams-compat.test.ts tests/dreams-anthropic.test.ts tests/dreams-bridge.test.ts tests/sentiment-tagging.test.ts tests/adaptive-sleep.test.ts tests/epistemic-confidence.test.ts tests/explainable-memory.test.ts tests/causal-graph.test.ts tests/abstractions.test.ts tests/bootstrap.test.ts tests/contested-conflicts.test.ts tests/benchmark-metrics.test.ts", "test:multi-device": "node --import tsx/esm --test tests/multi-device.test.ts", "test:sentiment-tagging": "node --import tsx/esm --test tests/sentiment-tagging.test.ts", "test:adaptive-sleep": "node --import tsx/esm --test tests/adaptive-sleep.test.ts", @@ -55,6 +55,7 @@ "test:abstractions": "node --import tsx/esm --test tests/abstractions.test.ts", "test:bootstrap": "node --import tsx/esm --test tests/bootstrap.test.ts", "test:contested-conflicts": "node --import tsx/esm --test tests/contested-conflicts.test.ts", + "test:benchmark-metrics": "node --import tsx/esm --test tests/benchmark-metrics.test.ts", "benchmark:longmemeval": "node --import tsx/esm benchmarks/longmemeval/run.ts", "benchmark:download": "node --import tsx/esm benchmarks/longmemeval/download.ts", "benchmark:ingest": "node --import tsx/esm benchmarks/longmemeval/ingest.ts", diff --git a/tests/benchmark-metrics.test.ts b/tests/benchmark-metrics.test.ts new file mode 100644 index 0000000..b9f8414 --- /dev/null +++ b/tests/benchmark-metrics.test.ts @@ -0,0 +1,163 @@ +// MemForge — benchmark scoring metric tests +// +// These exist because the published "93.2% R@5" was produced by a scorer with +// no test coverage, and it was not R@5: evaluate.ts called +// recallAtK(ids, answers, ids.length), so the slice never truncated and every +// k measured the same full candidate list. Consolidation packs many sessions +// into one warm row, so that list could hold hundreds of sessions. +// +// The guards below pin the distinction that made the old number wrong: a +// session-level metric must respect k, and the row-level metric must be +// reported separately rather than presented as R@k. +// +// Run: node --import tsx/esm --test tests/benchmark-metrics.test.ts +// Requires: no database. + +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; + +import { + extractSessionIds, + recallAtKSessions, + recallAtKRows, + percentile, + latencyStats, +} from '../benchmarks/lib/metrics.js'; + +// ─── extractSessionIds ─────────────────────────────────────────────────────── + +describe('extractSessionIds', () => { + it('pulls every marker in order of appearance', () => { + const content = '[SESSION_ID:7] USER: hi\n\n[SESSION_ID:9] USER: bye'; + + assert.deepEqual(extractSessionIds(content), ['7', '9']); + }); + + it('returns an empty array when no markers are present', () => { + assert.deepEqual(extractSessionIds('plain warm-tier content'), []); + }); + + it('is repeatable across calls', () => { + // The regex is module-level and /g-flagged. Reading it via exec without + // resetting lastIndex made the second call skip matches; matchAll does + // not share that state. Pinned because the failure is silent — the + // scorer would quietly under-count retrieved sessions. + const content = '[SESSION_ID:1] a [SESSION_ID:2] b'; + + const first = extractSessionIds(content); + const second = extractSessionIds(content); + + assert.deepEqual(first, ['1', '2']); + assert.deepEqual(second, first, 'a second call must see the same markers'); + }); +}); + +// ─── recallAtKSessions ─────────────────────────────────────────────────────── + +describe('recallAtKSessions', () => { + const retrieved = ['10', '11', '12', '13', '14', '15']; + + it('scores a hit when the gold session is inside the first k', () => { + assert.equal(recallAtKSessions(retrieved, ['12'], 3), 1); + }); + + it('scores a miss when the gold session falls outside the first k', () => { + // THE REGRESSION GUARD. The previous scorer passed the array's own length + // as k, so the slice never truncated and this case returned 1 — inflating + // every reported R@1/R@3/R@5. + assert.equal(recallAtKSessions(retrieved, ['15'], 3), 0); + }); + + it('respects k=1 strictly', () => { + assert.equal(recallAtKSessions(retrieved, ['10'], 1), 1); + assert.equal(recallAtKSessions(retrieved, ['11'], 1), 0); + }); + + it('is monotonic in k', () => { + // A larger window can only help; a scorer that ignores k would return the + // same value everywhere, which this would not catch alone — hence the + // strict miss case above. + const at1 = recallAtKSessions(retrieved, ['13'], 1); + const at5 = recallAtKSessions(retrieved, ['13'], 5); + + assert.equal(at1, 0); + assert.equal(at5, 1); + }); + + it('matches numeric gold ids against string session ids', () => { + assert.equal(recallAtKSessions(['42'], [42 as unknown as string], 1), 1); + }); + + it('scores a miss with no gold sessions or no retrievals', () => { + assert.equal(recallAtKSessions(retrieved, [], 5), 0); + assert.equal(recallAtKSessions([], ['1'], 5), 0); + }); +}); + +// ─── recallAtKRows ─────────────────────────────────────────────────────────── + +describe('recallAtKRows', () => { + // Row 0 packs three sessions — the shape consolidation actually produces. + const perRow = [['1', '2', '3'], ['4', '5'], ['6']]; + + it('scores a hit for any gold session inside the top-k rows', () => { + assert.equal(recallAtKRows(perRow, ['3'], 1), 1, 'session 3 is packed into row 0'); + }); + + it('excludes rows beyond k', () => { + assert.equal(recallAtKRows(perRow, ['6'], 2), 0, 'session 6 lives in row 2'); + assert.equal(recallAtKRows(perRow, ['6'], 3), 1); + }); + + it('is at least as generous as the session metric under packing', () => { + // This inequality is the whole reason both numbers are reported: with 3 + // sessions per row, R@1 over rows can hit where R@1 over sessions cannot. + const flat = perRow.flat(); + + assert.equal(recallAtKSessions(flat, ['3'], 1), 0); + assert.equal(recallAtKRows(perRow, ['3'], 1), 1); + }); + + it('converges with the session metric when rows hold one session each', () => { + const unpacked = [['1'], ['2'], ['3']]; + + for (const k of [1, 2, 3]) { + assert.equal( + recallAtKRows(unpacked, ['2'], k), + recallAtKSessions(unpacked.flat(), ['2'], k), + `metrics must agree at k=${k} when packing is 1:1`, + ); + } + }); +}); + +// ─── latency helpers ───────────────────────────────────────────────────────── + +describe('percentile', () => { + it('returns 0 for an empty set', () => { + assert.equal(percentile([], 95), 0); + }); + + it('picks the expected element of a sorted set', () => { + const sorted = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]; + + assert.equal(percentile(sorted, 50), 5); + assert.equal(percentile(sorted, 100), 10); + }); +}); + +describe('latencyStats', () => { + it('reports zeroes for an empty set rather than NaN', () => { + const stats = latencyStats([]); + + assert.deepEqual(stats, { p50: 0, p95: 0, p99: 0, mean: 0, min: 0, max: 0 }); + }); + + it('computes mean, min and max over unsorted input', () => { + const stats = latencyStats([30, 10, 20]); + + assert.equal(stats.mean, 20); + assert.equal(stats.min, 10); + assert.equal(stats.max, 30); + }); +});