A50: Exclude locally-initiated client cancellations from Call Counter (#12923) - #558
A50: Exclude locally-initiated client cancellations from Call Counter (#12923)#558AgraVator wants to merge 8 commits into
Conversation
|
|
||
| ### Excluding Locally-Initiated Client-Side Cancellations | ||
|
|
||
| Locally-initiated client-side cancellations (hedging cancellations, application `CANCELLED`, and `DEADLINE_EXCEEDED`) are excluded from outlier detection counting (`successCount` and `failureCount`). For locally-initiated cancellations (hedging cancellations of non-winning sibling attempts and application-initiated cancellations), this aligns with Envoy's `resetStream()` behavior and prevents false-positive ejections during hedging. For `DEADLINE_EXCEEDED`, this differs from Envoy (which counts deadline expirations as failures for outlier detection); however, because deadline expiration semantics vary between Envoy and gRPC, and because a deadline expiration cannot be definitively attributed to a server failure rather than a network delay, gRPC treats it as a client-side cancellation. |
There was a problem hiding this comment.
A race or network delay can result in client trigger CANCELLED due to deadline before server responds with DEADLINE_EXCEEDED. This can result in outlier detector not counting it. Aren't both scenarios same and counted equally? Ref: #560
There was a problem hiding this comment.
Hi,
Client timeouts can stem from client-side queuing or transit delays rather than endpoint unhealthiness which should not be counted as failures.
cc @ejona86
There was a problem hiding this comment.
Please let me know if my understanding is correct: once a physical attempt has been assigned to an endpoint, the client deadline can race with the server’s DEADLINE_EXCEEDED response. The same attempt may therefore be excluded as a client-side timeout or counted as a server response depending on which signal arrives first. Is the proposal intentionally excluding client/parent deadline expiry while counting a server-returned DEADLINE_EXCEEDED? We are hoping the API can expose enough information for a policy to optionally count deadline expiry for an active endpoint attempt, while still excluding explicit cancellations and non-winning retry or hedging attempts.
Summary
Clarifies gRFC A50 (
A50-xds-outlier-detection.md) to explicitly specify that locally-initiated client-side cancellations (hedging cancellations, application cancellation, and deadline exceeded) are excluded from the Outlier Detection Call Counter.Key Clarifications
resetStream()behavior (silently resetting the stream without recording endpoint errors), preventing false-positive ejections during hedging.Related Work