Skip to content

feat: ValorBrain memory provider + BEAM-100K result (71.4% avg score) - #32

Draft
agentxagi wants to merge 1 commit into
vectorize-io:mainfrom
ValorBrain:valorbrain-submission
Draft

feat: ValorBrain memory provider + BEAM-100K result (71.4% avg score)#32
agentxagi wants to merge 1 commit into
vectorize-io:mainfrom
ValorBrain:valorbrain-submission

Conversation

@agentxagi

Copy link
Copy Markdown

What

Adds ValorBrain as a memory provider, plus the LLM adapters needed to reproduce our reader/judge pair, and our complete BEAM-100K run.

Result (beam / 100k / rag — 400/400 queries)

Accuracy (avg score per criterion) 71.4%
Binary correct 304/400 (76.0%)
Reader Gemini 3.6 Flash
Judge GLM-5.2 (different model family from reader)
Avg retrieve time 5.2s
Avg context tokens 12.6k

The provider

Hybrid memory engine for agents — BM25 + dense vectors (pgvector) + RRF + entity-graph reranking + cross-encoder rerank, on PostgreSQL with row-level security. Retrieval goes through the production path agents use at runtime (/api/v1/memory/prepare), with re-windowed ingest (100k chunks → 8k windows w/ overlap) matching production conversational windows. Plain-HTTP REST, no SDK dependency.

Reproducibility notes

  • Full run JSON: outputs/beam/valorbrain/rag/100k.json (answer_llm, judge_llm, per-query scores and judge reasons included)
  • Judge override: we pin GLM-5.2 as judge (different family than the Gemini reader) via OMB_JUDGE_LLM — the override mechanism is part of this PR (runner.py), documented in the README
  • The score was independently reproduced with a completely different reader (stealth "Ox Alpha" via OpenCode Zen): 70.2% avg score / 76.0% binary on the same 400 queries — evidence the number measures the memory system, not one reader
  • Engine: self-hosted; benchmark tenant configuration in the README

Checklist

  • Provider registered in the memory registry
  • 400/400 queries, no partials
  • Judge from a different model family than the reader
  • Results JSON + manifest entry included

Happy to adjust to whatever process you prefer for leaderboard inclusion (external_results.json entry, re-run under your infra, etc.).

… 400 queries)

Adds the ValorBrain memory provider (hybrid retrieval over REST, re-windowed
ingest), the LLM providers needed to reproduce our reader/judge pair
(gateway/glm/agy-direct, OpenAI-compatible plain-HTTP with reasoning-model
fallback), and our complete BEAM-100K run: 400/400 queries, 71.4% avg score
per criterion (304/400 = 76.0% binary correct), reader Gemini 3.6 Flash,
judge GLM-5.2 (different family from the reader), avg retrieve 5.2s, avg
context 12.6k tokens. Reproduced independently with a different reader
(70.2%). Full results in outputs/beam/valorbrain/rag/100k.json.
@vercel

vercel Bot commented Aug 21, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the Vectorize Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant