Skip to content

perf: fix keyword search returning ≤1 result and Redis-absent stalls - #179

Merged
salishforge merged 1 commit into
mainfrom
perf/retrieval-and-cache-fixes
Jul 28, 2026
Merged

perf: fix keyword search returning ≤1 result and Redis-absent stalls#179
salishforge merged 1 commit into
mainfrom
perf/retrieval-and-cache-fixes

Conversation

@salishforge

Copy link
Copy Markdown
Owner

Found by running the corrected benchmark. Both are real defects, not tuning knobs — and neither trades quality, security, or functionality for a number.

1. Keyword search matched almost nothing

queryKeyword used plainto_tsquery, which ANDs every lexeme. "what kind of exercise did I say I do" becomes 'kind' & 'exercis' & 'say', and a single conversation session rarely contains all of them.

Measured on the real corpus: AND matched 0 rows where OR matched 16. Keyword mode returned a median of one result for any natural-language question — and since hybrid fuses the two arms, hybrid was effectively running semantic-only.

Rewriting the operators to OR restores graded matching, with ts_rank_cd discriminating: rows matching more query lexemes, closer together, outrank weak single-term hits. Precision comes from ranking + LIMIT rather than from refusing to match. The rewrite runs through plainto_tsquery's own output, so stemming, stop-words and escaping stay Postgres's job — no user input reaches query syntax.

Recall@k (sessions) before after
keyword R@3 60.0% 83.3%
keyword R@5 60.0% 92.9%
keyword R@10 60.0% 95.2%

Latency cost: p50 5ms → 9ms.

2. An absent Redis stalled every cached read

A failed connect costs ~7.5s (connectTimeout plus the reconnect ladder), and the attempt left no memory of itself — so /query and /stats re-paid it on every request, while the docs advertise Redis as optional with graceful degradation.

Circuit breaker: after a failure, skip Redis until a cooldown expires, then allow one probe. Consecutive queries measured 7.6s → 0.016s → 0.007s. Behaviour with Redis present is unchanged.

Benchmark methodology fixes found along the way

  • Limited runs were category-biased. LongMemEval-S stores instances in contiguous blocks by question type, so BENCHMARK_LIMIT=50 sampled 50 single-session-user questions and nothing else. Tuning retrieval against that would have optimised for whichever category sat at the offset. Adds deterministic stratified sampling (default for partial runs; BENCHMARK_SAMPLE=sequential restores the old slice).
  • Partial runs could masquerade as official. The generator overwrites RESULTS.md in place, so a 20-question smoke test silently replaced the published page in the official format. Partial runs now carry a not-an-official-result banner naming sample size and strategy.

Stratified baseline (n=42, 7 per category)

keyword hybrid
R@1 69.0% 69.0%
R@5 92.9% 92.9%
R@10 95.2% 92.9%
p50 9ms 28ms

Deliberately not done

Two things I measured but did not tune, because n=42 is too small to justify moving a constant and doing so would be fitting to the benchmark:

  • Hybrid R@10 (92.9%) is below keyword alone (95.2%). Semantic's 1.5× RRF weight can crowd out correct keyword hits. That weight was calibrated when the keyword arm returned ~1 result, so it likely wants revisiting — but on a full run, not this sample.
  • single-session-preference is the weakest category (71.4% keyword / 85.7% hybrid), independently confirming WB-03 in NEXTGEN-RECOMMENDATIONS.md.

Also noted: tests/memory-namespaces.test.ts hangs — verified pre-existing on main, and it is absent from the npm test list, which is why it went unnoticed.

Full suite green: integration 24, http 43, security 29, epistemic 33, causal 39, abstractions 39, bootstrap 22, contested 9, metrics 17, cache-degradation 5.

🤖 Generated with Claude Code

https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru

Two defects found by running the corrected benchmark, both real bugs
rather than tuning knobs.

1. Keyword search matched almost nothing. queryKeyword used
   plainto_tsquery, which ANDs every lexeme: "what kind of exercise did
   I say I do" becomes 'kind' & 'exercis' & 'say', and a single
   conversation session rarely contains all of them. Measured on the
   LongMemEval corpus the AND form matched 0 rows where OR matched 16 —
   keyword mode returned a median of ONE result for any natural-language
   question, and because hybrid fuses the two arms, hybrid was
   effectively running semantic-only.

   Rewriting the operators to OR restores graded matching, with
   ts_rank_cd doing the discriminating: rows matching more query lexemes,
   closer together, rank above weak single-term hits. Precision comes
   from ranking plus LIMIT rather than from refusing to match. The
   rewrite goes through plainto_tsquery's own output, so stemming,
   stop-words and escaping remain Postgres's job and no user input
   reaches query syntax.

   Stratified 42-question run, Recall@k over sessions:
     keyword  R@3 60.0% → 83.3%,  R@5 60.0% → 92.9%,  R@10 60.0% → 95.2%
   Latency cost p50 5ms → 9ms.

2. An absent Redis stalled every cached read. A failed connect costs
   ~7.5s (connectTimeout plus the reconnect ladder) and the attempt left
   no memory of itself, so /query and /stats re-paid it on every single
   request — while the docs advertised Redis as optional with graceful
   degradation. Adds a circuit breaker: after a failure, skip Redis until
   a cooldown expires, then allow one probe. Measured on the benchmark
   server, consecutive queries went 7.6s → 0.016s → 0.007s. Behaviour
   with Redis present is unchanged.

Benchmark methodology fixes found along the way:

- LongMemEval-S stores instances in contiguous blocks by question type,
  so BENCHMARK_LIMIT=50 sampled 50 single-session-user questions and
  nothing else. Every limited run was category-biased, and tuning
  retrieval against one would have optimised for whichever category sat
  at the offset. Adds deterministic stratified sampling (default for
  partial runs; BENCHMARK_SAMPLE=sequential restores the old slice).

- The report generator overwrites RESULTS.md in place, so a 20-question
  smoke test silently replaced the published page — formatted exactly
  like an official result. Partial runs now carry a prominent
  not-an-official-result banner naming the sample size and strategy.

Tests: tests/cache-degradation.test.ts covers the no-Redis path, which
had none because the existing cache suite exits early when Redis is
missing — that gap is why this shipped. No test relaxed or removed; full
suite green (integration 24, http 43, security 29, epistemic 33, causal
39, abstractions 39, bootstrap 22, contested 9, metrics 17).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru
@salishforge
salishforge merged commit 6863fd7 into main Jul 28, 2026
14 checks passed
@salishforge
salishforge deleted the perf/retrieval-and-cache-fixes branch July 28, 2026 01:24
salishforge added a commit that referenced this pull request Jul 29, 2026
…ic 1.5:1

The 1.5 semantic weight was calibrated while the keyword arm was returning
roughly one result per query — the plainto_tsquery AND-semantics bug fixed in
#179. It was fit against a broken arm and never re-derived once that arm
worked, so the ratio encoded a defect rather than a measurement.

Swept {0.5, 0.75, 1.0, 1.25, 1.5, 2.0} across all 500 LongMemEval questions
against a split and objective registered before the sweep ran. 1.0 wins:

  weight   R@1     R@5     R@10
  1.5      83.2%   95.2%   97.0%   (previous default)
  1.0      83.8%   95.4%   97.6%

The gains are small — one to three questions — but they replicate on both
halves of a stratified split for R@1, R@5 and R@10 (R@3 flips sign, noted).
The stronger argument is the guardrail: fusion should never bury a session that
a single arm ranks first. At 1.5 that happened in single-session-user on 5.7%
of questions, matching the QA regression in that category to the decimal; at
1.0 it halves to 1.4%, and single-session-assistant stops regressing too.
Dropping below 1.0 reverses the problem and starts costing the categories where
the vector arm is decisive.

1.0 is also the principled default — standard RRF weights its arms equally, so
this removes a thumb on the scale rather than adding one. HYBRID_SEMANTIC_WEIGHT
still overrides; the optimum is embedding-model dependent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant