feat: ValorBrain memory provider + BEAM-100K result (71.4% avg score) - #32
Draft
agentxagi wants to merge 1 commit into
Draft
feat: ValorBrain memory provider + BEAM-100K result (71.4% avg score)#32agentxagi wants to merge 1 commit into
agentxagi wants to merge 1 commit into
Conversation
… 400 queries) Adds the ValorBrain memory provider (hybrid retrieval over REST, re-windowed ingest), the LLM providers needed to reproduce our reader/judge pair (gateway/glm/agy-direct, OpenAI-compatible plain-HTTP with reasoning-model fallback), and our complete BEAM-100K run: 400/400 queries, 71.4% avg score per criterion (304/400 = 76.0% binary correct), reader Gemini 3.6 Flash, judge GLM-5.2 (different family from the reader), avg retrieve 5.2s, avg context 12.6k tokens. Reproduced independently with a different reader (70.2%). Full results in outputs/beam/valorbrain/rag/100k.json.
|
Someone is attempting to deploy a commit to the Vectorize Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds ValorBrain as a memory provider, plus the LLM adapters needed to reproduce our reader/judge pair, and our complete BEAM-100K run.
Result (beam / 100k / rag — 400/400 queries)
The provider
Hybrid memory engine for agents — BM25 + dense vectors (pgvector) + RRF + entity-graph reranking + cross-encoder rerank, on PostgreSQL with row-level security. Retrieval goes through the production path agents use at runtime (
/api/v1/memory/prepare), with re-windowed ingest (100k chunks → 8k windows w/ overlap) matching production conversational windows. Plain-HTTP REST, no SDK dependency.Reproducibility notes
outputs/beam/valorbrain/rag/100k.json(answer_llm,judge_llm, per-query scores and judge reasons included)OMB_JUDGE_LLM— the override mechanism is part of this PR (runner.py), documented in the READMEChecklist
Happy to adjust to whatever process you prefer for leaderboard inclusion (external_results.json entry, re-run under your infra, etc.).