feat: Phase 3 — semantic cache (local embeddings + per-tenant)

.env.example

@@ -22,9 +22,16 @@ TRACE_DB_PATH=./traces.db
 # Optional: also export OpenTelemetry spans to an OTLP/HTTP collector (e.g. Jaeger).
 # OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces
 
+# ── Semantic cache (embeds prompts locally; serves similar repeats) ───
+CACHE_ENABLED=true
+# Cosine-similarity threshold for a cache hit (0–1; higher = stricter).
+CACHE_SIMILARITY_THRESHOLD=0.92
+CACHE_TTL_SECONDS=3600
+CACHE_MAX_ENTRIES=1000
+
 # ── Local models via Ollama (keyless — no quota, no rate limit) ───────
-# OpenAI-compatible base URL of YOUR Ollama (used for local completions,
-# and — in a later phase — the local judge + embeddings).
+# OpenAI-compatible base URL of YOUR Ollama (used for local completions and the
+# semantic-cache embeddings via EMBED_MODEL; the judge reuses it in a later phase).
 OLLAMA_BASE_URL=http://localhost:11434/v1
 JUDGE_MODEL=qwen2.5:7b
 EMBED_MODEL=nomic-embed-text

README.md

@@ -2,7 +2,7 @@
 
 A self-hostable **verifying LLM gateway** — a drop-in, OpenAI-compatible proxy that **routes** (cheapest capable model + fallback), **semantically caches**, and **verifies** (deterministic guardrails inline + a local Ollama judge) every LLM call, with full OpenTelemetry tracing. Unlike after-the-fact observability tools, it can flag or block a bad response _before it returns_.
 
-> 🚧 **Early development.** Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md). Currently at **Phase 2 — tracing & persistence** (routing, caching, and verification land in later phases).
+> 🚧 **Early development.** Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md). Currently at **Phase 3 — semantic cache** (routing/fallback and verification land in later phases).
 
 ## What works today (Phase 1)
 
@@ -97,6 +97,10 @@ curl http://localhost:8080/traces \
 
 Traces are **metadata only** — no prompt or response bodies are stored, and API keys are recorded as a SHA-256 hash, never in the clear.
 
+## Caching
+
+Sentinel **semantically caches** responses: it embeds each prompt locally (Ollama `nomic-embed-text`) and, when a new request is similar enough to a recent one (cosine ≥ `CACHE_SIMILARITY_THRESHOLD`, default `0.92`), serves the stored answer **without calling the provider** — replaying buffered SSE chunks for streamed requests. The cache is **per-tenant** (scoped to the calling API key), bounded (`CACHE_MAX_ENTRIES`, `CACHE_TTL_SECONDS`), and **fails open** — any embedding error simply falls through to the provider. Cache hits are visible in traces (`GET /traces?cacheHit=true`). Disable with `CACHE_ENABLED=false`.
+
 ## Development
 
 ```bash

SECURITY_REVIEW_LOG.md

@@ -38,8 +38,8 @@ Each item: assume an attacker is actively trying it. Tick only when a test prove
 
 ### 5. Cache poisoning / data leakage
 
-- [ ] Cache key includes everything that changes the answer (model, system prompt, params) — a different context cannot collide into a wrong hit.
-- [ ] No cross-tenant cache hits; the similarity threshold cannot leak another key's data.
+- [x] Cache key includes everything that changes the answer (model, params, stream, embed-model) — a different context cannot collide into a wrong hit.
+- [x] No cross-tenant cache hits (entries bucketed by API-key hash); the similarity threshold cannot leak another key's data.
 
 ### 6. Log / trace data hygiene
 
@@ -69,5 +69,6 @@ Each item: assume an attacker is actively trying it. Tick only when a test prove
 | —      | 2026-06-27 | —          | Initial context docs (no code) | n/a                                                                          | n/a      | n/a       | Baseline.                                                                                                                                                                                                                                                                                                                                                                                         |
 | SR-001 | 2026-06-27 | 1, 3, 4    | Phase 1 pass-through proxy     | Unauthenticated access; key/secret leakage; SSRF via request-controlled URLs | high     | mitigated | Bearer Sentinel-key auth required — 401 on missing/invalid (tested). Provider keys read from env via `apiKeyEnv`, never hard-coded, never returned to clients. Provider base URLs come only from the config file, never request input. `authorization`/`x-api-key` redacted in request logs via pino `redact` (configured; an explicit log-assertion test is still TODO — box 1.2 left unticked). |
 | SR-002 | 2026-06-27 | 3, 6       | Phase 2 tracing & persistence  | Unauthorized trace access; secrets/PII in stored traces                      | high     | mitigated | `GET /traces` gated by a separate `SENTINEL_ADMIN_KEY` — 401 without it (tested), distinct from client keys. Traces are metadata-only (model, provider, tokens, latency, status) — no prompt/response bodies persisted. API keys stored as a SHA-256 hash, never raw. Per-key trace scoping deferred (admin-only for now).                                                                        |
+| SR-003 | 2026-06-27 | 5          | Phase 3 semantic cache         | Cross-tenant cache leakage; wrong-answer collisions                          | high     | mitigated | Cache entries are bucketed per-tenant by API-key hash — no cross-tenant hits (tested). The bucket also keys on model/temperature/max_tokens/stream/embed-model, so requests that change the answer never collide; semantic matching happens only within a bucket above a conservative threshold (0.92, configurable). Cache fails open (embed errors → miss).                                     |
 
 > Add a row per high-risk change. Status ∈ {open, mitigated, accepted}. Severity ∈ {low, med, high, critical}.

packages/gateway/src/cache/cache.test.ts

@@ -0,0 +1,93 @@
+import { describe, it, expect } from 'vitest';
+import { createSemanticCache } from './cache.js';
+import type { Embedder } from './embedder.js';
+import type { ChatCompletionRequest } from '../schemas.js';
+
+function req(content: string, over: Partial<ChatCompletionRequest> = {}): ChatCompletionRequest {
+  return { model: 'm', messages: [{ role: 'user', content }], ...over };
+}
+
+function embedderFrom(fn: (text: string) => number[]): Embedder {
+  return { embed: (text) => Promise.resolve(fn(text)) };
+}
+
+const base = { threshold: 0.9, ttlMs: 1000, maxEntries: 10, embedModel: 'e' };
+
+describe('createSemanticCache', () => {
+  it('hits a semantically similar request and misses a dissimilar one', async () => {
+    const vec = (t: string): number[] =>
+      t.includes('bye') ? [0, 1] : t.includes('there') ? [0.99, 0.14] : [1, 0];
+    const cache = createSemanticCache({ embedder: embedderFrom(vec), ...base });
+    await cache.set(req('hi'), 'key1', { kind: 'json', body: { answer: 1 } });
+    expect(await cache.get(req('hi there'), 'key1')).toEqual({ kind: 'json', body: { answer: 1 } });
+    expect(await cache.get(req('bye'), 'key1')).toBeUndefined();
+  });
+
+  it('does not hit across tenants', async () => {
+    const cache = createSemanticCache({ embedder: embedderFrom(() => [1, 0]), ...base });
+    await cache.set(req('hi'), 'key1', { kind: 'json', body: 1 });
+    expect(await cache.get(req('hi'), 'key2')).toBeUndefined();
+  });
+
+  it('does not hit across models', async () => {
+    const cache = createSemanticCache({ embedder: embedderFrom(() => [1, 0]), ...base });
+    await cache.set(req('hi', { model: 'a' }), 'key1', { kind: 'json', body: 1 });
+    expect(await cache.get(req('hi', { model: 'b' }), 'key1')).toBeUndefined();
+  });
+
+  it('keeps stream and non-stream entries in separate buckets', async () => {
+    const cache = createSemanticCache({ embedder: embedderFrom(() => [1, 0]), ...base });
+    await cache.set(req('hi', { stream: true }), 'key1', { kind: 'stream', chunks: ['a'] });
+    expect(await cache.get(req('hi'), 'key1')).toBeUndefined();
+    expect(await cache.get(req('hi', { stream: true }), 'key1')).toEqual({
+      kind: 'stream',
+      chunks: ['a'],
+    });
+  });
+
+  it('expires entries past their TTL', async () => {
+    let t = 1000;
+    const cache = createSemanticCache({
+      embedder: embedderFrom(() => [1, 0]),
+      threshold: 0.9,
+      ttlMs: 100,
+      maxEntries: 10,
+      embedModel: 'e',
+      now: () => t,
+    });
+    await cache.set(req('hi'), 'key1', { kind: 'json', body: 1 });
+    t = 1050;
+    expect(await cache.get(req('hi'), 'key1')).toEqual({ kind: 'json', body: 1 });
+    t = 1200;
+    expect(await cache.get(req('hi'), 'key1')).toBeUndefined();
+  });
+
+  it('evicts the oldest entry beyond maxEntries', async () => {
+    const vecByContent: Record<string, number[]> = { q1: [1, 0, 0], q2: [0, 1, 0], q3: [0, 0, 1] };
+    const embed = embedderFrom((text) => {
+      const c = text.includes('q1') ? 'q1' : text.includes('q2') ? 'q2' : 'q3';
+      return vecByContent[c]!;
+    });
+    const cache = createSemanticCache({
+      embedder: embed,
+      threshold: 0.99,
+      ttlMs: 10000,
+      maxEntries: 2,
+      embedModel: 'e',
+    });
+    await cache.set(req('q1'), 'k', { kind: 'json', body: 1 });
+    await cache.set(req('q2'), 'k', { kind: 'json', body: 2 });
+    await cache.set(req('q3'), 'k', { kind: 'json', body: 3 });
+    expect(await cache.get(req('q1'), 'k')).toBeUndefined();
+    expect(await cache.get(req('q3'), 'k')).toEqual({ kind: 'json', body: 3 });
+  });
+
+  it('fails open when the embedder throws', async () => {
+    const cache = createSemanticCache({
+      embedder: { embed: () => Promise.reject(new Error('down')) },
+      ...base,
+    });
+    await expect(cache.set(req('hi'), 'k', { kind: 'json', body: 1 })).resolves.toBeUndefined();
+    expect(await cache.get(req('hi'), 'k')).toBeUndefined();
+  });
+});

packages/gateway/src/cache/embedder.test.ts

@@ -0,0 +1,41 @@
+import { describe, it, expect, vi } from 'vitest';
+import { createOllamaEmbedder } from './embedder.js';
+
+describe('createOllamaEmbedder', () => {
+  it('posts to /embeddings and returns the vector', async () => {
+    const fetchImpl = vi.fn(async (_url: string, _init: { body: string }) =>
+      Promise.resolve(
+        new Response(JSON.stringify({ data: [{ embedding: [0.1, 0.2, 0.3] }] }), { status: 200 }),
+      ),
+    );
+    const embedder = createOllamaEmbedder({
+      baseUrl: 'http://h/v1/',
+      model: 'nomic-embed-text',
+      fetchImpl,
+    });
+
+    const vec = await embedder.embed('hello');
+    expect(vec).toEqual([0.1, 0.2, 0.3]);
+
+    const call = fetchImpl.mock.calls[0]!;
+    expect(call[0]).toBe('http://h/v1/embeddings');
+    expect(JSON.parse(call[1].body) as { model: string; input: string }).toEqual({
+      model: 'nomic-embed-text',
+      input: 'hello',
+    });
+  });
+
+  it('throws on a non-OK response', async () => {
+    const fetchImpl = vi.fn(() => Promise.resolve(new Response('nope', { status: 500 })));
+    const embedder = createOllamaEmbedder({ baseUrl: 'http://h', model: 'm', fetchImpl });
+    await expect(embedder.embed('x')).rejects.toThrow();
+  });
+
+  it('throws when the embedding is missing', async () => {
+    const fetchImpl = vi.fn(() =>
+      Promise.resolve(new Response(JSON.stringify({ data: [] }), { status: 200 })),
+    );
+    const embedder = createOllamaEmbedder({ baseUrl: 'http://h', model: 'm', fetchImpl });
+    await expect(embedder.embed('x')).rejects.toThrow();
+  });
+});

packages/gateway/src/cache/embedder.ts

@@ -0,0 +1,50 @@
+import type { FetchLike } from '../providers/types.js';
+
+/** Produces an embedding vector for a piece of text. */
+export interface Embedder {
+  embed(text: string): Promise<number[]>;
+}
+
+export interface OllamaEmbedderOptions {
+  baseUrl: string;
+  model: string;
+  apiKey?: string | undefined;
+  /** Injectable fetch (defaults to global `fetch`); handy for tests. */
+  fetchImpl?: FetchLike;
+}
+
+interface EmbeddingsResponse {
+  data?: { embedding?: unknown }[];
+}
+
+/**
+ * Embeds text via an OpenAI-compatible `/embeddings` endpoint — e.g. local Ollama
+ * `nomic-embed-text` (keyless). Mirrors the provider adapter's fetch/header pattern.
+ */
+export function createOllamaEmbedder(options: OllamaEmbedderOptions): Embedder {
+  const fetchImpl: FetchLike = options.fetchImpl ?? fetch;
+  const endpoint = `${options.baseUrl.replace(/\/+$/, '')}/embeddings`;
+
+  return {
+    async embed(text: string): Promise<number[]> {
+      const headers: Record<string, string> = { 'content-type': 'application/json' };
+      if (options.apiKey !== undefined && options.apiKey.length > 0) {
+        headers.authorization = `Bearer ${options.apiKey}`;
+      }
+      const res = await fetchImpl(endpoint, {
+        method: 'POST',
+        headers,
+        body: JSON.stringify({ model: options.model, input: text }),
+      });
+      if (!res.ok) {
+        throw new Error(`embeddings request failed: HTTP ${res.status}`);
+      }
+      const json = (await res.json()) as EmbeddingsResponse;
+      const embedding = json.data?.[0]?.embedding;
+      if (!Array.isArray(embedding)) {
+        throw new Error('embeddings response missing data[0].embedding');
+      }
+      return embedding.map((n) => (typeof n === 'number' ? n : 0));
+    },
+  };
+}

packages/gateway/src/cache/vector.test.ts

@@ -0,0 +1,21 @@
+import { describe, it, expect } from 'vitest';
+import { cosineSimilarity } from './vector.js';
+
+describe('cosineSimilarity', () => {
+  it('is 1 for identical vectors', () => {
+    expect(cosineSimilarity([1, 2, 3], [1, 2, 3])).toBeCloseTo(1);
+  });
+
+  it('is 0 for orthogonal vectors', () => {
+    expect(cosineSimilarity([1, 0], [0, 1])).toBe(0);
+  });
+
+  it('returns 0 for length mismatch or empty input', () => {
+    expect(cosineSimilarity([1, 2], [1])).toBe(0);
+    expect(cosineSimilarity([], [])).toBe(0);
+  });
+
+  it('returns 0 for a zero vector', () => {
+    expect(cosineSimilarity([0, 0], [1, 1])).toBe(0);
+  });
+});

packages/gateway/src/cache/vector.ts

@@ -0,0 +1,16 @@
+/** Cosine similarity of two equal-length vectors. Returns 0 for mismatched lengths or a zero vector. */
+export function cosineSimilarity(a: readonly number[], b: readonly number[]): number {
+  if (a.length === 0 || a.length !== b.length) return 0;
+  let dot = 0;
+  let normA = 0;
+  let normB = 0;
+  for (let i = 0; i < a.length; i++) {
+    const x = a[i] ?? 0;
+    const y = b[i] ?? 0;
+    dot += x * y;
+    normA += x * x;
+    normB += y * y;
+  }
+  if (normA === 0 || normB === 0) return 0;
+  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
+}

packages/gateway/src/config.test.ts

@@ -46,6 +46,28 @@ describe('loadServerEnv', () => {
     expect(env.traceDb).toBe('sqlite');
     expect(env.adminKey).toBeUndefined();
   });
+
+  it('parses cache settings with sensible defaults', () => {
+    const def = loadServerEnv({ SENTINEL_API_KEYS: 'k' });
+    expect(def.cacheEnabled).toBe(true);
+    expect(def.cacheThreshold).toBe(0.92);
+    expect(def.ollamaBaseUrl).toBe('http://localhost:11434/v1');
+    expect(def.embedModel).toBe('nomic-embed-text');
+
+    const custom = loadServerEnv({
+      SENTINEL_API_KEYS: 'k',
+      CACHE_ENABLED: 'false',
+      CACHE_SIMILARITY_THRESHOLD: '0.8',
+      CACHE_TTL_SECONDS: '60',
+      CACHE_MAX_ENTRIES: '5',
+      EMBED_MODEL: 'custom-embed',
+    });
+    expect(custom.cacheEnabled).toBe(false);
+    expect(custom.cacheThreshold).toBe(0.8);
+    expect(custom.cacheTtlSeconds).toBe(60);
+    expect(custom.cacheMaxEntries).toBe(5);
+    expect(custom.embedModel).toBe('custom-embed');
+  });
 });
 
 const validConfig = JSON.stringify({

packages/gateway/src/config.ts

@@ -11,6 +11,12 @@ export interface ServerEnv {
   adminKey: string | undefined;
   traceDb: 'sqlite' | 'memory';
   traceDbPath: string;
+  cacheEnabled: boolean;
+  cacheThreshold: number;
+  cacheTtlSeconds: number;
+  cacheMaxEntries: number;
+  ollamaBaseUrl: string;
+  embedModel: string;
 }
 
 const serverEnvSchema = z.object({
@@ -20,6 +26,15 @@ const serverEnvSchema = z.object({
   SENTINEL_ADMIN_KEY: z.string().optional(),
   TRACE_DB: z.enum(['sqlite', 'memory']).default('sqlite'),
   TRACE_DB_PATH: z.string().default('./traces.db'),
+  CACHE_ENABLED: z
+    .enum(['true', 'false'])
+    .default('true')
+    .transform((v) => v === 'true'),
+  CACHE_SIMILARITY_THRESHOLD: z.coerce.number().min(0).max(1).default(0.92),
+  CACHE_TTL_SECONDS: z.coerce.number().int().positive().default(3600),
+  CACHE_MAX_ENTRIES: z.coerce.number().int().positive().default(1000),
+  OLLAMA_BASE_URL: z.string().default('http://localhost:11434/v1'),
+  EMBED_MODEL: z.string().default('nomic-embed-text'),
 });
 
 /** Reads and validates the process environment Sentinel needs to run. */
@@ -41,6 +56,12 @@ export function loadServerEnv(env: NodeJS.ProcessEnv): ServerEnv {
     adminKey: parsed.data.SENTINEL_ADMIN_KEY,
     traceDb: parsed.data.TRACE_DB,
     traceDbPath: parsed.data.TRACE_DB_PATH,
+    cacheEnabled: parsed.data.CACHE_ENABLED,
+    cacheThreshold: parsed.data.CACHE_SIMILARITY_THRESHOLD,
+    cacheTtlSeconds: parsed.data.CACHE_TTL_SECONDS,
+    cacheMaxEntries: parsed.data.CACHE_MAX_ENTRIES,
+    ollamaBaseUrl: parsed.data.OLLAMA_BASE_URL,
+    embedModel: parsed.data.EMBED_MODEL,
   };
 }
 

packages/gateway/src/index.ts

@@ -20,3 +20,7 @@ export {
 } from './errors.js';
 export { createTraceStore } from './telemetry/store.js';
 export type { TraceStore, TraceRecord, TraceQuery } from './telemetry/trace.js';
+export { createSemanticCache } from './cache/cache.js';
+export type { SemanticCache, CachedResponse } from './cache/cache.js';
+export { createOllamaEmbedder } from './cache/embedder.js';
+export type { Embedder } from './cache/embedder.js';

packages/gateway/src/main.ts

@@ -5,6 +5,9 @@ import { buildServer } from './server.js';
 import { ConfigError } from './errors.js';
 import { createTraceStore } from './telemetry/store.js';
 import { initTelemetry } from './telemetry/otel.js';
+import { createOllamaEmbedder } from './cache/embedder.js';
+import { createSemanticCache } from './cache/cache.js';
+import type { SemanticCache } from './cache/cache.js';
 
 async function main(): Promise<void> {
   const env = loadServerEnv(process.env);
@@ -14,11 +17,24 @@ async function main(): Promise<void> {
   const shutdownTelemetry = initTelemetry(store, {
     otlpEndpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
   });
+
+  let cache: SemanticCache | undefined;
+  if (env.cacheEnabled) {
+    cache = createSemanticCache({
+      embedder: createOllamaEmbedder({ baseUrl: env.ollamaBaseUrl, model: env.embedModel }),
+      threshold: env.cacheThreshold,
+      ttlMs: env.cacheTtlSeconds * 1000,
+      maxEntries: env.cacheMaxEntries,
+      embedModel: env.embedModel,
+    });
+  }
+
   const app = buildServer({
     registry,
     apiKeys: env.apiKeys,
     traceStore: store,
     adminKey: env.adminKey,
+    cache,
   });
 
   const shutdown = async (): Promise<void> => {