perf: fix keyword search returning ≤1 result and Redis-absent stalls - #179
Merged
Conversation
Two defects found by running the corrected benchmark, both real bugs
rather than tuning knobs.
1. Keyword search matched almost nothing. queryKeyword used
plainto_tsquery, which ANDs every lexeme: "what kind of exercise did
I say I do" becomes 'kind' & 'exercis' & 'say', and a single
conversation session rarely contains all of them. Measured on the
LongMemEval corpus the AND form matched 0 rows where OR matched 16 —
keyword mode returned a median of ONE result for any natural-language
question, and because hybrid fuses the two arms, hybrid was
effectively running semantic-only.
Rewriting the operators to OR restores graded matching, with
ts_rank_cd doing the discriminating: rows matching more query lexemes,
closer together, rank above weak single-term hits. Precision comes
from ranking plus LIMIT rather than from refusing to match. The
rewrite goes through plainto_tsquery's own output, so stemming,
stop-words and escaping remain Postgres's job and no user input
reaches query syntax.
Stratified 42-question run, Recall@k over sessions:
keyword R@3 60.0% → 83.3%, R@5 60.0% → 92.9%, R@10 60.0% → 95.2%
Latency cost p50 5ms → 9ms.
2. An absent Redis stalled every cached read. A failed connect costs
~7.5s (connectTimeout plus the reconnect ladder) and the attempt left
no memory of itself, so /query and /stats re-paid it on every single
request — while the docs advertised Redis as optional with graceful
degradation. Adds a circuit breaker: after a failure, skip Redis until
a cooldown expires, then allow one probe. Measured on the benchmark
server, consecutive queries went 7.6s → 0.016s → 0.007s. Behaviour
with Redis present is unchanged.
Benchmark methodology fixes found along the way:
- LongMemEval-S stores instances in contiguous blocks by question type,
so BENCHMARK_LIMIT=50 sampled 50 single-session-user questions and
nothing else. Every limited run was category-biased, and tuning
retrieval against one would have optimised for whichever category sat
at the offset. Adds deterministic stratified sampling (default for
partial runs; BENCHMARK_SAMPLE=sequential restores the old slice).
- The report generator overwrites RESULTS.md in place, so a 20-question
smoke test silently replaced the published page — formatted exactly
like an official result. Partial runs now carry a prominent
not-an-official-result banner naming the sample size and strategy.
Tests: tests/cache-degradation.test.ts covers the no-Redis path, which
had none because the existing cache suite exits early when Redis is
missing — that gap is why this shipped. No test relaxed or removed; full
suite green (integration 24, http 43, security 29, epistemic 33, causal
39, abstractions 39, bootstrap 22, contested 9, metrics 17).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru
salishforge
added a commit
that referenced
this pull request
Jul 29, 2026
…ic 1.5:1 The 1.5 semantic weight was calibrated while the keyword arm was returning roughly one result per query — the plainto_tsquery AND-semantics bug fixed in #179. It was fit against a broken arm and never re-derived once that arm worked, so the ratio encoded a defect rather than a measurement. Swept {0.5, 0.75, 1.0, 1.25, 1.5, 2.0} across all 500 LongMemEval questions against a split and objective registered before the sweep ran. 1.0 wins: weight R@1 R@5 R@10 1.5 83.2% 95.2% 97.0% (previous default) 1.0 83.8% 95.4% 97.6% The gains are small — one to three questions — but they replicate on both halves of a stratified split for R@1, R@5 and R@10 (R@3 flips sign, noted). The stronger argument is the guardrail: fusion should never bury a session that a single arm ranks first. At 1.5 that happened in single-session-user on 5.7% of questions, matching the QA regression in that category to the decimal; at 1.0 it halves to 1.4%, and single-session-assistant stops regressing too. Dropping below 1.0 reverses the problem and starts costing the categories where the vector arm is decisive. 1.0 is also the principled default — standard RRF weights its arms equally, so this removes a thumb on the scale rather than adding one. HYBRID_SEMANTIC_WEIGHT still overrides; the optimum is embedding-model dependent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by running the corrected benchmark. Both are real defects, not tuning knobs — and neither trades quality, security, or functionality for a number.
1. Keyword search matched almost nothing
queryKeywordusedplainto_tsquery, which ANDs every lexeme. "what kind of exercise did I say I do" becomes'kind' & 'exercis' & 'say', and a single conversation session rarely contains all of them.Measured on the real corpus: AND matched 0 rows where OR matched 16. Keyword mode returned a median of one result for any natural-language question — and since hybrid fuses the two arms, hybrid was effectively running semantic-only.
Rewriting the operators to OR restores graded matching, with
ts_rank_cddiscriminating: rows matching more query lexemes, closer together, outrank weak single-term hits. Precision comes from ranking + LIMIT rather than from refusing to match. The rewrite runs throughplainto_tsquery's own output, so stemming, stop-words and escaping stay Postgres's job — no user input reaches query syntax.Latency cost: p50 5ms → 9ms.
2. An absent Redis stalled every cached read
A failed connect costs ~7.5s (connectTimeout plus the reconnect ladder), and the attempt left no memory of itself — so
/queryand/statsre-paid it on every request, while the docs advertise Redis as optional with graceful degradation.Circuit breaker: after a failure, skip Redis until a cooldown expires, then allow one probe. Consecutive queries measured 7.6s → 0.016s → 0.007s. Behaviour with Redis present is unchanged.
Benchmark methodology fixes found along the way
BENCHMARK_LIMIT=50sampled 50single-session-userquestions and nothing else. Tuning retrieval against that would have optimised for whichever category sat at the offset. Adds deterministic stratified sampling (default for partial runs;BENCHMARK_SAMPLE=sequentialrestores the old slice).RESULTS.mdin place, so a 20-question smoke test silently replaced the published page in the official format. Partial runs now carry a not-an-official-result banner naming sample size and strategy.Stratified baseline (n=42, 7 per category)
Deliberately not done
Two things I measured but did not tune, because n=42 is too small to justify moving a constant and doing so would be fitting to the benchmark:
single-session-preferenceis the weakest category (71.4% keyword / 85.7% hybrid), independently confirming WB-03 inNEXTGEN-RECOMMENDATIONS.md.Also noted:
tests/memory-namespaces.test.tshangs — verified pre-existing onmain, and it is absent from thenpm testlist, which is why it went unnoticed.Full suite green: integration 24, http 43, security 29, epistemic 33, causal 39, abstractions 39, bootstrap 22, contested 9, metrics 17, cache-degradation 5.
🤖 Generated with Claude Code
https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru