Skip to content

Cache the LiteLLM provider in the router instead of resolving per-request #397

Description

Problem.
router/router.py resolves provider config lazily but may re-instantiate per request. Provider construction has measurable overhead at high request rates.

Actions.

  • Audit the LiteLLM provider construction path; confirm caching status.
  • If not cached, memoize on (agent_id, provider_config_hash).
  • Add a microbenchmark in tests/perf/ to lock the improvement in.

Acceptance:
provider is constructed once per (agent_id, config) for the lifetime of a run.

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions