Add SSE keepalive comments to prevent connection idle timeouts#121
Closed
adambalogh wants to merge 1 commit into
Closed
Add SSE keepalive comments to prevent connection idle timeouts#121adambalogh wants to merge 1 commit into
adambalogh wants to merge 1 commit into
Conversation
…timeouts
gpt-image-2 generations routinely run 60-120s, during which the streaming
chat response carried zero bytes: the image is only emitted on the final SSE
frame. Intermediaries between the browser and the enclave (OHTTP relay, load
balancers, the browser's HTTP/2 session) kill connections idle that long,
surfacing client-side as ERR_HTTP2_PROTOCOL_ERROR at ~60s.
Run the blocking provider call in a worker thread and yield SSE comment
lines (': keepalive') every 10s while it runs. Comments are ignored by SSE
parsers per spec (the chat-app client already skips non-'data:' lines), and
the OHTTP controller seals each yielded chunk as it arrives, so keepalives
keep every hop's idle timers reset without changing the response contract.
Applied to both blocking image paths: the endpoint-based streaming responder
(gpt-image-2, grok-2-image, Seedream/Seedance, GLM-Image) and the Gemini
inline-image invoke in the chat streaming controller.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uhw2GhoEnxG98rmbgpHhhP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Image generation requests can block for 60-120 seconds with no bytes on the wire, causing intermediaries (OHTTP relay, load balancers, browser HTTP/2 sessions) to kill the connection as idle at ~60s. This change adds SSE keepalive comment streaming during long-running provider calls to keep the connection alive while maintaining full backward compatibility with existing response handling.
Key Changes
New
run_with_sse_keepalive()generator function inimage_generation.py:ThreadPoolExecutor: keepalive\n\n) at configurable intervals (default 10s) while the function executesUpdated image generation endpoints to use keepalive streaming:
create_image_generation_streaming_response()now wraps_run_image_generation()withrun_with_sse_keepalive()chat_controller.pywraps image output model invocation with keepalive streamingComprehensive test coverage in
TestSseKeepalive:Implementation Details
ThreadPoolExecutorwithwait=Falseshutdown ensures background provider calls complete even if client disconnectshttps://claude.ai/code/session_01Uhw2GhoEnxG98rmbgpHhhP