feat: Phase 2 — request tracing & persistence (OpenTelemetry + SQLite)

.env.example

@@ -12,6 +12,15 @@ SENTINEL_API_KEYS=dev-local-key
 PORT=8080
 # Path to the provider config file (defaults to ./sentinel.config.json)
 SENTINEL_CONFIG=./sentinel.config.json
+# Admin key that gates the GET /traces observability API (leave blank to disable it).
+SENTINEL_ADMIN_KEY=
+
+# ── Tracing & persistence ────────────────────────────────────────────
+# Where request traces are stored: "sqlite" (default) or "memory".
+TRACE_DB=sqlite
+TRACE_DB_PATH=./traces.db
+# Optional: also export OpenTelemetry spans to an OTLP/HTTP collector (e.g. Jaeger).
+# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces
 
 # ── Local models via Ollama (keyless — no quota, no rate limit) ───────
 # OpenAI-compatible base URL of YOUR Ollama (used for local completions,

README.md

@@ -2,7 +2,7 @@
 
 A self-hostable **verifying LLM gateway** — a drop-in, OpenAI-compatible proxy that **routes** (cheapest capable model + fallback), **semantically caches**, and **verifies** (deterministic guardrails inline + a local Ollama judge) every LLM call, with full OpenTelemetry tracing. Unlike after-the-fact observability tools, it can flag or block a bad response _before it returns_.
 
-> 🚧 **Early development.** Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md). Currently at **Phase 1 — the pass-through proxy** (routing, caching, and verification land in later phases).
+> 🚧 **Early development.** Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md). Currently at **Phase 2 — tracing & persistence** (routing, caching, and verification land in later phases).
 
 ## What works today (Phase 1)
 
@@ -83,6 +83,20 @@ Because almost every provider speaks the OpenAI API, **bringing your own key is
 
 Sentinel never hardcodes a machine or a model. Set `OLLAMA_BASE_URL` in `.env` to point at _your_ Ollama (default `http://localhost:11434/v1`), list the models you've pulled in the `models` map, and you're set. The local **judge** that verifies outputs in a later phase works the same way — it reads `JUDGE_MODEL` from your environment, so every clone uses its own local model, with no API key and no quota.
 
+## Observability
+
+Every request is traced with OpenTelemetry — provider, model, status, latency, and token usage — and persisted to a queryable store (SQLite by default; set `TRACE_DB=memory` for an ephemeral one). Spans also export to any OTLP collector (Jaeger, etc.) when `OTEL_EXPORTER_OTLP_ENDPOINT` is set.
+
+Read recent traces through the admin-gated API (set `SENTINEL_ADMIN_KEY` to enable it):
+
+```bash
+curl http://localhost:8080/traces \
+  -H "authorization: Bearer $SENTINEL_ADMIN_KEY"
+# filters: ?model=gpt-4o-mini&status=200&stream=true&since=<epoch-ms>&limit=50
+```
+
+Traces are **metadata only** — no prompt or response bodies are stored, and API keys are recorded as a SHA-256 hash, never in the clear.
+
 ## Development
 
 ```bash

SECURITY_REVIEW_LOG.md

@@ -29,7 +29,7 @@ Each item: assume an attacker is actively trying it. Tick only when a test prove
 
 - [x] Every request authenticated by a Sentinel API key; invalid/missing ⇒ 401.
 - [ ] Cache entries and traces namespaced per key; cross-tenant read is impossible (tested).
-- [ ] Admin/config endpoints separated from proxy traffic and authorized distinctly.
+- [x] Admin/config endpoints separated from proxy traffic and authorized distinctly.
 
 ### 4. SSRF / outbound calls
 
@@ -43,7 +43,7 @@ Each item: assume an attacker is actively trying it. Tick only when a test prove
 
 ### 6. Log / trace data hygiene
 
-- [ ] PII/prompt redaction policy applied before persistence (configurable; on by default for sensitive fields).
+- [x] Traces are metadata-only — prompt/response bodies are never persisted; API keys stored as a SHA-256 hash, never raw.
 - [ ] Stored trace content is escaped on render in the dashboard (no stored XSS).
 
 ### 7. Availability / DoS
@@ -68,5 +68,6 @@ Each item: assume an attacker is actively trying it. Tick only when a test prove
 | ------ | ---------- | ---------- | ------------------------------ | ---------------------------------------------------------------------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | —      | 2026-06-27 | —          | Initial context docs (no code) | n/a                                                                          | n/a      | n/a       | Baseline.                                                                                                                                                                                                                                                                                                                                                                                         |
 | SR-001 | 2026-06-27 | 1, 3, 4    | Phase 1 pass-through proxy     | Unauthenticated access; key/secret leakage; SSRF via request-controlled URLs | high     | mitigated | Bearer Sentinel-key auth required — 401 on missing/invalid (tested). Provider keys read from env via `apiKeyEnv`, never hard-coded, never returned to clients. Provider base URLs come only from the config file, never request input. `authorization`/`x-api-key` redacted in request logs via pino `redact` (configured; an explicit log-assertion test is still TODO — box 1.2 left unticked). |
+| SR-002 | 2026-06-27 | 3, 6       | Phase 2 tracing & persistence  | Unauthorized trace access; secrets/PII in stored traces                      | high     | mitigated | `GET /traces` gated by a separate `SENTINEL_ADMIN_KEY` — 401 without it (tested), distinct from client keys. Traces are metadata-only (model, provider, tokens, latency, status) — no prompt/response bodies persisted. API keys stored as a SHA-256 hash, never raw. Per-key trace scoping deferred (admin-only for now).                                                                        |
 
 > Add a row per high-risk change. Status ∈ {open, mitigated, accepted}. Severity ∈ {low, med, high, critical}.

package.json

@@ -38,7 +38,8 @@
   },
   "pnpm": {
     "onlyBuiltDependencies": [
-      "esbuild"
+      "esbuild",
+      "better-sqlite3"
     ]
   }
 }

packages/gateway/package.json

@@ -5,8 +5,19 @@
   "type": "module",
   "description": "Sentinel gateway — the verifying LLM proxy core. (Scaffold; built out from ROADMAP Phase 1.)",
   "dependencies": {
+    "@opentelemetry/api": "^1.9.1",
+    "@opentelemetry/core": "^2.8.0",
+    "@opentelemetry/exporter-trace-otlp-http": "^0.219.0",
+    "@opentelemetry/resources": "^2.8.0",
+    "@opentelemetry/sdk-trace-base": "^2.8.0",
+    "@opentelemetry/sdk-trace-node": "^2.8.0",
+    "@opentelemetry/semantic-conventions": "^1.41.1",
+    "better-sqlite3": "^11.10.0",
     "dotenv": "^17.4.2",
     "fastify": "^5.8.5",
     "zod": "^3.25.76"
+  },
+  "devDependencies": {
+    "@types/better-sqlite3": "^7.6.13"
   }
 }

packages/gateway/src/auth.ts

@@ -1,3 +1,4 @@
+import { createHash } from 'node:crypto';
 import { AuthError } from './errors.js';
 
 /** Extracts the bearer token from an Authorization header, or null if absent/malformed. */
@@ -22,3 +23,18 @@ export function createAuthHook(apiKeys: ReadonlySet<string>) {
     }
   };
 }
+
+/** Builds a preHandler that requires the configured admin key (for trace/admin endpoints). */
+export function createAdminAuthHook(adminKey: string | undefined) {
+  return async function adminAuthHook(request: RequestWithAuth): Promise<void> {
+    const token = extractBearerToken(request.headers.authorization);
+    if (adminKey === undefined || adminKey.length === 0 || token !== adminKey) {
+      throw new AuthError('Admin access required');
+    }
+  };
+}
+
+/** SHA-256 hex digest of an API key, so raw keys are never persisted in traces. */
+export function hashApiKey(key: string): string {
+  return createHash('sha256').update(key).digest('hex');
+}

packages/gateway/src/config.test.ts

@@ -28,6 +28,24 @@ describe('loadServerEnv', () => {
   it('throws when PORT is not a positive integer', () => {
     expect(() => loadServerEnv({ SENTINEL_API_KEYS: 'k', PORT: 'abc' })).toThrow(ConfigError);
   });
+
+  it('reads admin key and trace DB settings', () => {
+    const env = loadServerEnv({
+      SENTINEL_API_KEYS: 'k',
+      SENTINEL_ADMIN_KEY: 'admin',
+      TRACE_DB: 'memory',
+      TRACE_DB_PATH: '/tmp/t.db',
+    });
+    expect(env.adminKey).toBe('admin');
+    expect(env.traceDb).toBe('memory');
+    expect(env.traceDbPath).toBe('/tmp/t.db');
+  });
+
+  it('defaults trace DB to sqlite and admin key to undefined', () => {
+    const env = loadServerEnv({ SENTINEL_API_KEYS: 'k' });
+    expect(env.traceDb).toBe('sqlite');
+    expect(env.adminKey).toBeUndefined();
+  });
 });
 
 const validConfig = JSON.stringify({

packages/gateway/src/config.ts

@@ -8,12 +8,18 @@ export interface ServerEnv {
   port: number;
   apiKeys: ReadonlySet<string>;
   configPath: string;
+  adminKey: string | undefined;
+  traceDb: 'sqlite' | 'memory';
+  traceDbPath: string;
 }
 
 const serverEnvSchema = z.object({
   PORT: z.coerce.number().int().positive().default(8080),
   SENTINEL_API_KEYS: z.string().default(''),
   SENTINEL_CONFIG: z.string().default('./sentinel.config.json'),
+  SENTINEL_ADMIN_KEY: z.string().optional(),
+  TRACE_DB: z.enum(['sqlite', 'memory']).default('sqlite'),
+  TRACE_DB_PATH: z.string().default('./traces.db'),
 });
 
 /** Reads and validates the process environment Sentinel needs to run. */
@@ -32,6 +38,9 @@ export function loadServerEnv(env: NodeJS.ProcessEnv): ServerEnv {
     port: parsed.data.PORT,
     apiKeys: new Set(apiKeys),
     configPath: parsed.data.SENTINEL_CONFIG,
+    adminKey: parsed.data.SENTINEL_ADMIN_KEY,
+    traceDb: parsed.data.TRACE_DB,
+    traceDbPath: parsed.data.TRACE_DB_PATH,
   };
 }
 

packages/gateway/src/index.ts

@@ -18,3 +18,5 @@ export {
   UpstreamError,
   ConfigError,
 } from './errors.js';
+export { createTraceStore } from './telemetry/store.js';
+export type { TraceStore, TraceRecord, TraceQuery } from './telemetry/trace.js';

packages/gateway/src/main.ts

@@ -3,12 +3,32 @@ import { loadServerEnv, loadConfig } from './config.js';
 import { createRegistry } from './providers/registry.js';
 import { buildServer } from './server.js';
 import { ConfigError } from './errors.js';
+import { createTraceStore } from './telemetry/store.js';
+import { initTelemetry } from './telemetry/otel.js';
 
 async function main(): Promise<void> {
   const env = loadServerEnv(process.env);
   const config = loadConfig({ path: env.configPath, env: process.env });
   const registry = createRegistry(config);
-  const app = buildServer({ registry, apiKeys: env.apiKeys });
+  const store = createTraceStore({ kind: env.traceDb, path: env.traceDbPath });
+  const shutdownTelemetry = initTelemetry(store, {
+    otlpEndpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
+  });
+  const app = buildServer({
+    registry,
+    apiKeys: env.apiKeys,
+    traceStore: store,
+    adminKey: env.adminKey,
+  });
+
+  const shutdown = async (): Promise<void> => {
+    await app.close();
+    await shutdownTelemetry();
+    store.close();
+  };
+  process.on('SIGTERM', () => void shutdown());
+  process.on('SIGINT', () => void shutdown());
+
   await app.listen({ port: env.port, host: '0.0.0.0' });
 }
 

packages/gateway/src/routes.traces.ts

@@ -0,0 +1,45 @@
+import type { FastifyPluginAsync } from 'fastify';
+import { createAdminAuthHook } from './auth.js';
+import type { TraceQuery, TraceStore } from './telemetry/trace.js';
+
+export interface TraceRoutesOptions {
+  traceStore: TraceStore;
+  adminKey: string | undefined;
+}
+
+/** Fastify plugin: an admin-key-gated read API over the trace store. */
+export const traceRoutes: FastifyPluginAsync<TraceRoutesOptions> = async (app, options) => {
+  const adminHook = createAdminAuthHook(options.adminKey);
+
+  app.get('/traces', { preHandler: adminHook }, async (request, reply) => {
+    return reply.send(options.traceStore.query(parseTraceQuery(request.query)));
+  });
+
+  app.get('/traces/:id', { preHandler: adminHook }, async (request, reply) => {
+    const { id } = request.params as { id: string };
+    const trace = options.traceStore.get(id);
+    if (trace === undefined) {
+      return reply
+        .status(404)
+        .send({ error: { message: `No trace "${id}"`, type: 'not_found', code: null } });
+    }
+    return reply.send(trace);
+  });
+};
+
+function parseTraceQuery(raw: unknown): TraceQuery {
+  const q = (typeof raw === 'object' && raw !== null ? raw : {}) as Record<
+    string,
+    string | undefined
+  >;
+  const query: TraceQuery = {};
+  if (q.model !== undefined) query.model = q.model;
+  if (q.provider !== undefined) query.provider = q.provider;
+  if (q.status !== undefined) query.status = Number(q.status);
+  if (q.stream !== undefined) query.stream = q.stream === 'true';
+  if (q.since !== undefined) query.since = Number(q.since);
+  if (q.until !== undefined) query.until = Number(q.until);
+  if (q.limit !== undefined) query.limit = Math.min(Number(q.limit), 500);
+  if (q.offset !== undefined) query.offset = Number(q.offset);
+  return query;
+}