Skip to content

feat(ai-proxy): support phase-specific HTTP timeouts - #13801

Draft
ruanweihong-aaa wants to merge 2 commits into
apache:masterfrom
ruanweihong-aaa:feat/ai-proxy-phase-timeouts
Draft

feat(ai-proxy): support phase-specific HTTP timeouts#13801
ruanweihong-aaa wants to merge 2 commits into
apache:masterfrom
ruanweihong-aaa:feat/ai-proxy-phase-timeouts

Conversation

@ruanweihong-aaa

@ruanweihong-aaa ruanweihong-aaa commented Aug 10, 2026

Copy link
Copy Markdown

Description

ai-proxy and ai-proxy-multi currently expose a single timeout value that is applied to the HTTP connect, send, and read phases. This makes it difficult to fail fast during connection establishment or request upload while allowing a longer timeout for LLM response generation.

This PR adds three optional phase-specific timeout fields:

  • connect_timeout: timeout for establishing the upstream connection;
  • send_timeout: timeout for sending the request to the upstream;
  • read_timeout: timeout for each socket read from the upstream.

All three fields are expressed in milliseconds and accept integer values from 1 to 600000.

The change preserves backward compatibility:

  • when none of the new fields is configured, the transport continues to call the existing numeric set_timeout(timeout) path;
  • when only some phase-specific fields are configured, every omitted phase falls back to the existing timeout value;
  • existing configurations therefore keep their previous behavior.

Implementation details:

  • add the three fields to both the ai-proxy and ai-proxy-multi schemas;
  • build a complete connect/send/read timeout tuple at the ai-proxy transport call site;
  • use lua-resty-http's set_timeouts(connect, send, read) for phase-specific configuration while retaining the original set_timeout(timeout) compatibility path;
  • document the new fields in the English and Chinese plugin documentation;
  • add schema boundary tests, transport API tests, and delayed-upstream read-timeout regression coverage.

Compatibility

The new fields are optional. Existing users who only configure timeout, or rely on its default value, continue to use the original code path. The upper bound of each new field is the same as the existing timeout upper bound.

read_timeout is a per-socket-read timeout and is not a total wall-clock limit for streaming responses. Existing max_stream_duration_ms and max_response_bytes controls remain responsible for total streaming duration and response-size limits.

Verification

Targeted, source-matched Docker tests:

  • schema validation for ai-proxy and ai-proxy-multi, including valid boundaries and invalid values;
  • transport dispatch to set_timeouts(connect, send, read) and compatibility with the numeric set_timeout(timeout) path;
  • delayed-upstream read-timeout behavior with only read_timeout configured, exercising fallback to the existing timeout value for the omitted phases;
  • result: Files=3, Tests=63, Result: PASS.

Additional local dual-Gateway A/B verification used identical etcd, routes, Docker network, and controlled upstream fixtures for the pre-fix and fixed images:

  • read phase: pre-fix returned 200 after about 250 ms; fixed returned 504 after about 50 ms with read_timeout=50;
  • connect phase: pre-fix timed out after about 1 s; fixed timed out after about 50 ms with connect_timeout=50;
  • send phase: with a 60 MiB request and an upstream that accepted headers but did not consume the body, the fixed image timed out significantly earlier with send_timeout=50;
  • Gateway logs were checked to associate each failure with the intended connect, send, or read phase.

These results cover the tests related to this change; they are not a claim that the complete APISIX test suite was executed locally.

Which issue(s) this PR fixes:

Fixes #12072

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible

Add optional connect, send, and read timeout settings for ai-proxy and ai-proxy-multi while preserving the existing timeout fallback behavior.
Comment thread t/plugin/ai-proxy-timeout-callsite.t Outdated

__DATA__

=== TEST 1: ai-proxy resolves phase timeout fallbacks at the transport call site

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These test cases implemented with mock code are of little value and can be removed.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the mock-only call-site test in 84c4343. The remaining coverage uses the plugin schema tests, the transport timeout tests, and the Test::Nginx delayed-upstream cases, including partial configuration with only read_timeout set. The focused suite passes: 3 files, 63 tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: more precise control over the timeout for requests to the LLM service

2 participants