Skip to content

perf: cache the OpenRouter model catalog with a 30-minute TTL - #90

Draft
aryansk wants to merge 1 commit into
shauryagangrade:mainfrom
aryansk:codex/issue-56-catalog-cache
Draft

perf: cache the OpenRouter model catalog with a 30-minute TTL#90
aryansk wants to merge 1 commit into
shauryagangrade:mainfrom
aryansk:codex/issue-56-catalog-cache

Conversation

@aryansk

@aryansk aryansk commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Problem

list_free_models hits https://openrouter.ai/api/v1/models on every call.
/models and startup both fetch the full catalog each time — a wasted
network round-trip and a slow menu on a flaky connection. Closes #56.

Change

Adds an in-process cache to list_free_models in gcode/models.py:

  • Successful responses are cached for 30 minutes (monotonic clock), so
    repeated calls inside the TTL make one HTTP request.
  • ttl_seconds=0 is the escape hatch for callers that need fresh data.
  • Failed fetches are never cached, so the next call retries; the
    best-effort contract (empty list + actionable error) is unchanged.
  • The returned list is a copy, so callers can't mutate the cache.
  • clear_model_catalog_cache() is exposed for tests/consumers.

Why this approach

In-process only, no new dependencies, and the default matches the issue's
suggested 10–60 min window. Only successful responses are cached so a
transient outage is never masked by a stale catalog, and the escape hatch
keeps /models honest when the user explicitly asks for a fresh list.

Testing

uv run pytest:  112 passed (5 new: cache reuse within TTL, real-clock
                expiry, ttl_seconds=0 bypass, failure-not-cached, copy
                isolation)
uv run ruff check .:  All checks passed
uv run ruff format --check .:  32 files already formatted
uv run mypy gcode:  Success: no issues found in 12 source files

Acceptance criteria covered: two consecutive calls inside the TTL make one
HTTP request (asserted via mock_get.call_count == 1), and cache
invalidation on expiry is tested with the real clock.

Documentation and release impact

  • User-facing documentation updated
  • Changelog/release note needed
  • Migration or compatibility note needed
  • No documentation impact

Review notes

  • Known limitations: none.
  • Follow-up issue: none.

Repeated list_free_models calls (startup + /models) inside the TTL now
reuse the last successful response instead of a network round-trip.
ttl_seconds=0 forces a fresh fetch; failed fetches are never cached.
Closes shauryagangrade#56.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance]: Cache the OpenRouter model catalog with a TTL

1 participant