Skip to content

A50: Exclude locally-initiated client cancellations from Call Counter (#12923) - #558

Open
AgraVator wants to merge 8 commits into
grpc:masterfrom
AgraVator:a50-client-cancellations
Open

A50: Exclude locally-initiated client cancellations from Call Counter (#12923)#558
AgraVator wants to merge 8 commits into
grpc:masterfrom
AgraVator:a50-client-cancellations

Conversation

@AgraVator

@AgraVator AgraVator commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Clarifies gRFC A50 (A50-xds-outlier-detection.md) to explicitly specify that locally-initiated client-side cancellations (hedging cancellations, application cancellation, and deadline exceeded) are excluded from the Outlier Detection Call Counter.

Key Clarifications

  1. Picker: Updates the counter in the map entry on request completion.
  2. Call Counter: Specifies that the active bucket tracks successes and failures while excluding locally-initiated client-side cancellations (hedging cancellations, application cancellation, and deadline exceeded etc).
  3. Rationale:
    • Hedging & Application Cancellations: Align with Envoy's resetStream() behavior (silently resetting the stream without recording endpoint errors), preventing false-positive ejections during hedging.
    • Deadline Exceeded: Differs from Envoy (which counts timeouts as failures for outlier detection). In gRPC, client-side deadline expiration cannot be reliably distinguished from other client cancellations cross-language, and client timeouts cannot be definitively attributed to an endpoint failure rather than network or client-side delays.

Related Work

Comment thread A50-xds-outlier-detection.md Outdated
@AgraVator
AgraVator requested a review from kannanjgithub August 3, 2026 07:06
@ejona86
ejona86 requested a review from murgatroid99 August 7, 2026 16:15
Comment thread A50-xds-outlier-detection.md Outdated
Comment thread A50-xds-outlier-detection.md Outdated
Comment thread A50-xds-outlier-detection.md Outdated
Comment thread A50-xds-outlier-detection.md Outdated
Comment thread A50-xds-outlier-detection.md Outdated

### Excluding Locally-Initiated Client-Side Cancellations

Locally-initiated client-side cancellations (hedging cancellations, application `CANCELLED`, and `DEADLINE_EXCEEDED`) are excluded from outlier detection counting (`successCount` and `failureCount`). For locally-initiated cancellations (hedging cancellations of non-winning sibling attempts and application-initiated cancellations), this aligns with Envoy's `resetStream()` behavior and prevents false-positive ejections during hedging. For `DEADLINE_EXCEEDED`, this differs from Envoy (which counts deadline expirations as failures for outlier detection); however, because deadline expiration semantics vary between Envoy and gRPC, and because a deadline expiration cannot be definitively attributed to a server failure rather than a network delay, gRPC treats it as a client-side cancellation.

@kishorekasi kishorekasi Aug 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A race or network delay can result in client trigger CANCELLED due to deadline before server responds with DEADLINE_EXCEEDED. This can result in outlier detector not counting it. Aren't both scenarios same and counted equally? Ref: #560

@AgraVator AgraVator Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,
Client timeouts can stem from client-side queuing or transit delays rather than endpoint unhealthiness which should not be counted as failures.
cc @ejona86

@kishorekasi kishorekasi Aug 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let me know if my understanding is correct: once a physical attempt has been assigned to an endpoint, the client deadline can race with the server’s DEADLINE_EXCEEDED response. The same attempt may therefore be excluded as a client-side timeout or counted as a server response depending on which signal arrives first. Is the proposal intentionally excluding client/parent deadline expiry while counting a server-returned DEADLINE_EXCEEDED? We are hoping the API can expose enough information for a policy to optionally count deadline expiry for an active endpoint attempt, while still excluding explicit cancellations and non-winning retry or hedging attempts.

@AgraVator
AgraVator requested a review from ejona86 August 19, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants