Skip to content

Allow outlier detection to distinguish client deadline expiry from other cancellations #560

Description

@kishorekasi

When a client call deadline expires before the server responds, ClientCall.Listener receives DEADLINE_EXCEEDED, but the subchannel's ClientStreamTracer receives CANCELLED.

The path appears to be:

  1. ClientCallImpl.CancellationHandler calls RetriableStream.cancel(DEADLINE_EXCEEDED).
  2. RetriableStream commits a NoopClientStream as the logical winner.
  3. The physical substream is cancelled with CANCELLED_BECAUSE_COMMITTED.
  4. The subchannel tracer receives that physical stream status as CANCELLED.

Relevant code:

If the server instead returns DEADLINE_EXCEEDED, the tracer receives DEADLINE_EXCEEDED directly.

Desired outlier-detection behavior
We would like to count both of these as endpoint failures:

  • The server returns DEADLINE_EXCEEDED.
  • The client deadline expires while waiting for that endpoint.

We would not like to count:

  • Explicit application cancellation.
  • Context cancellation unrelated to deadline expiry.
  • Cancellation of non-winning retry or hedging attempts.

At present, these locally initiated cases appear indistinguishable to an attempt-level ClientStreamTracer because they can all arrive as CANCELLED.

Question
Is there a supported way for a load-balancer policy or ClientStreamTracer to distinguish client deadline expiry from other locally initiated cancellations?

If not, would grpc-java consider exposing the cancellation origin, or otherwise preserving enough information for an outlier-detection policy to make this distinction?

I noticed that gRFC proposal #558 currently proposes excluding locally initiated deadline expiry from outlier-detection counting. Is that intended to be a fixed semantic, or could policies support counting deadline expiry when it is considered a useful endpoint-health signal?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions