Skip to content

fix(coding-agent): spread an unqualified find_models across providers - #922

Open
BILLKISHORE wants to merge 3 commits into
PrimeIntellect-ai:mainfrom
BILLKISHORE:fix/799-find-models-provider-spread
Open

fix(coding-agent): spread an unqualified find_models across providers#922
BILLKISHORE wants to merge 3 commits into
PrimeIntellect-ai:mainfrom
BILLKISHORE:fix/799-find-models-provider-spread

Conversation

@BILLKISHORE

@BILLKISHORE BILLKISHORE commented Aug 8, 2026

Copy link
Copy Markdown

Closes #799.

Problem

rlm.find_models() with no query scores every candidate the same, so ordering falls entirely to the
alphabetical tiebreak before truncation:

let score = normalizedQuery ? Number.POSITIVE_INFINITY : 0;   // empty query: everything ties at 0
...
.sort((a, b) => a.score - b.score || a.selector.localeCompare(b.selector))
.slice(0, limit)                                              // default limit 8

The bare call is the natural first move for an agent orienting itself before delegating, and it
returns the alphabetically first provider's models only. In the reported run that read as the
complete reachable set, so an orchestrator concluded only the anthropic family was natively
reachable and routed three delegations through shell-CLI fallbacks that were not needed. The failure
mode is a plausible subset rather than an error, so nothing downstream contradicts the inference.

Change

Unqualified searches now round-robin across providers, preserving each provider's own ordering. A
query-driven search is untouched and still ranks purely by relevance.

The result is that a default find_models() samples the catalog instead of paging it alphabetically,
so every configured provider is visible in the first response.

Scope

The issue also suggests returning a truncation marker or a provider summary. Both change the shape of
the response, and find_models is typed as list[RLMModel] in prime-agent-runtime/src/rlm/__init__.py,
where _model_from_payload validates each entry. Surfacing a marker means changing that return type,
which breaks existing callers, so this PR is limited to the ordering defect that produced the reported
misread. Happy to follow up on the response shape separately if you want it.

Tests

packages/coding-agent/test/suite/regressions/799-find-models-provider-spread.test.ts covers a
four-provider catalog with ten models each: that a bare search spans all four rather than one, that
per-provider ordering stays stable, that a limit above the catalog size returns everything, that a
real query is still relevance-ranked, and that a single-provider catalog is unchanged.

The first two fail before this change with expected Set{ 'anthropic' } to deeply equal Set{ 'anthropic', 'openai-codex', 'openrouter', 'zai' }.

Verified with npm run check plus the new test and the existing
4649-subagent-model-selection.test.ts on Node 22.22.3.

Note

Fix rlm.find_models() to spread results across providers when query is empty

  • Previously, an empty query returned an alphabetical slice dominated by the first provider; now results are interleaved round-robin across providers while preserving each provider's internal order.
  • Introduces spreadAcrossProviders in rlm-runtime.ts to group candidates by provider and interleave them; non-empty queries retain relevance-based ranking.
  • Adds a regression test suite in 799-find-models-provider-spread.test.ts covering edge cases including limits, single-provider catalogs, and real queries.
  • Behavioral Change: empty-query results now return a different ordering and provider distribution than before.

Macroscope summarized 257209b.

@BILLKISHORE

Copy link
Copy Markdown
Author

#855 predates this by about 21 hours and covers the same ordering defect, plus the page metadata and the #810 token-order fix. I missed it when I checked whether #799 was claimed.

That PR is the more complete answer. Its RLMModelPage list subclass also keeps existing callers working, which is a better result than my read that the return shape could not change without breaking them.

Leaving this open only in case the smaller change is useful in the meantime: it is limited to the ordering defect, touches three files, and is currently mergeable where #855 needs a rebase. If you would rather take #855 whole, say so and I will close this.

avion23 pushed a commit to avion23/prime-agent that referenced this pull request Aug 9, 2026
@BILLKISHORE
BILLKISHORE force-pushed the fix/799-find-models-provider-spread branch 2 times, most recently from 142ee51 to 68399ad Compare August 11, 2026 21:32
@BILLKISHORE
BILLKISHORE force-pushed the fix/799-find-models-provider-spread branch from 68399ad to 257209b Compare August 12, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rlm.find_models() returns an alphabetical head slice that reads as the full reachable set

1 participant