Skip to content

feat(query): make the hybrid semantic weight configurable (default unchanged) - #181

Merged
salishforge merged 1 commit into
mainfrom
feat/configurable-hybrid-weight
Jul 28, 2026
Merged

feat(query): make the hybrid semantic weight configurable (default unchanged)#181
salishforge merged 1 commit into
mainfrom
feat/configurable-hybrid-weight

Conversation

@salishforge

Copy link
Copy Markdown
Owner

What changed

The 1.5× multiplier on the semantic arm's RRF contribution was a hardcoded constant with a large, undocumented effect: with RRF K=60, it means semantic ranks 1–31 all outscore a keyword-only rank-1 hit — lexical matches the vector arm missed are close to invisible. Now HYBRID_SEMANTIC_WEIGHT, documented in the config reference.

The default is deliberately unchanged, and that is the finding.

The experiment, and why the answer is "leave it alone"

Pre-registered before running: sweep on a stratified 42-question sample (7 per category), then validate the winner on a disjoint 42-question sample selected the same way. Adopt only if it beats the 1.5 baseline on held-out R@5 without regressing R@1 by more than 2pp.

weight train R@5 test R@5 train R@1 test R@1
0.75 95.2% 88.1% 76.2% 76.2%
1.50 (current) 92.9% 88.1% 69.0% 81.0%

On train, 0.75 looked clearly better across every k — the monotonic trend was tidy enough to be convincing, and any single-sample tuning would have adopted it. It did not replicate. On held-out data R@5 is identical and R@1 is 4.8pp worse.

Every metric flips direction between the two samples. That's the signature of sampling noise, not signal: at n=42 one question moves a metric 2.4pp, and the observed gaps are one to two questions.

So the pre-registered rule fails on both conditions, and the weight stays at 1.5.

Why this is still worth merging

  • The constant is now measurable per deployment. The optimum plausibly depends on the embedding model — this was measured with local bge-small, and a stronger embedder would likely shift it.
  • The K=60 interaction is documented where someone changing it will see it.
  • Resolving the question properly needs the full 500-question run, and the knob is what makes that a config change rather than a code change.

Gates: type-check, lint, integration 24, http 43, retrieval-determinism 3, causal-graph 39 — all green.

🤖 Generated with Claude Code

https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru

…changed)

The 1.5x multiplier on the semantic arm's RRF contribution was a
hardcoded constant with a large, undocumented effect: with the standard
RRF constant K=60, it means semantic ranks 1 through 31 all outscore a
keyword-only rank-1 hit, so lexical matches the vector arm missed are
close to invisible. Exposed as HYBRID_SEMANTIC_WEIGHT so the balance can
be measured per deployment rather than assumed.

The default is deliberately NOT changed. A train/test experiment says
the current value is fine and that the apparent alternatives are noise:

  sweep on a stratified 42-question sample (7 per category), validated
  on a disjoint 42-question sample selected the same way

  weight   train R@5   test R@5   train R@1   test R@1
    0.75       95.2%      88.1%       76.2%      76.2%
    1.50       92.9%      88.1%       69.0%      81.0%

On train, 0.75 looked clearly better and would have been adopted by any
single-sample tuning. It did not replicate: R@5 identical on held-out
data and R@1 4.8pp worse. Every metric flips direction between the two
samples, which is the signature of sampling noise rather than signal —
at n=42 a single question moves a metric 2.4pp, and the observed gaps
are one to two questions.

Pre-registered rule (declared before the sweep ran): adopt only if the
winner beats the 1.5 baseline on the held-out sample at R@5 without
regressing R@1 by more than 2pp. It failed both conditions, so the
weight stays at 1.5.

Resolving this properly needs the full 500-question run; the knob is
what makes that measurement possible without a code change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru
@salishforge
salishforge merged commit 756ff7f into main Jul 28, 2026
14 checks passed
@salishforge
salishforge deleted the feat/configurable-hybrid-weight branch July 28, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant