Skip to content

Improve streaming timeout handling and keep-alive logic #539

Description

Summary

Reference PR: microsoft/Agents-for-net#950

StreamingResponse did not robustly handle channel-imposed streaming timeouts and keep-alive requirements. This was especially problematic for M365 Copilot (BizChat), which visually errors the stream ("Something went wrong") if it does not receive an update within ~45 seconds, and which enforces an overall streaming time limit. Teams likewise enforces a ~2 minute activity timeout after which the bot must switch from sending intermediate messages to updating the final activity.

This issue tracks the enhancements to make streaming resilient to these channel timeouts, add keep-alive ("working") notices, and cleanly distinguish user cancellations from timeouts.

Problems to be fixed

  • BizChat inactivity timeout: If no update was sent within ~45s, the BizChat stream would visually error. There was no keep-alive mechanism.
  • BizChat overall streaming timeout: No handling for exceeding the M365 streaming time limit; buffered content could be lost and the user left with a broken stream.
  • Teams activity timeout: When a channel timed out, the code kept sending new intermediate messages instead of updating the existing (final) activity.
  • Ambiguous cancellation vs timeout: A Teams ContentStreamNotAllowed error was always treated as a user cancellation, even when the underlying cause was "Content stream finished due to exceeded streaming time." (a timeout).
  • Timer re-entrancy: The interval timer callback could overlap itself if a send took longer than the interval, causing out-of-order or duplicated work.
  • No way to gracefully stop a stream but keep working: No API to notify the client the stream ended (e.g., due to timeout) while allowing the underlying operation to continue in non-streaming/async mode.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions