Skip to content

Add retry with backoff for rate limit errors (HTTP 429) #37

Description

@JNK234

Problem

When using high-frequency LLM calls (e.g., game demos that call llm:choose every tick), rate limit errors from providers cause immediate failures. The extension throws a RuntimeException with no retry logic.

Example error from GPT-5.4-mini:

Rate limit reached for gpt-5.4-mini in organization ... on tokens per min (TPM): Limit 200000, Used 192306, Requested 11670. Please try again in 1.192s.

Current Behavior

BaseHttpProvider.sendChatRequest throws immediately on any HTTP error response, including rate limits.

Proposed Fix

Add retry with exponential backoff in BaseHttpProvider.sendChatRequest:

  • Detect rate limit responses (HTTP 429 or error body containing rate_limit)
  • Retry up to 3 times with exponential backoff (1s, 2s, 4s)
  • Parse Retry-After header if present and use that as the wait time
  • Only retry on rate limit errors — other errors should still fail immediately

Affected File

src/main/providers/BaseHttpProvider.scalasendChatRequest method (line ~103)

Workaround

Demos can handle this at the NetLogo level with wait + stop in the error handler:

carefully [
  let action llm:choose obs choices
  ...
] [
  if member? "rate_limit" error-message [
    wait 2
    stop
  ]
]

Context

Discovered while testing the Baba Is AI demo with gpt-5.4-mini. History accumulation + frequent calls hits TPM limits quickly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions