When a client call deadline expires before the server responds, ClientCall.Listener receives DEADLINE_EXCEEDED, but the subchannel's ClientStreamTracer receives CANCELLED.
The path appears to be:
- ClientCallImpl.CancellationHandler calls RetriableStream.cancel(
DEADLINE_EXCEEDED).
- RetriableStream commits a NoopClientStream as the logical winner.
- The physical substream is cancelled with
CANCELLED_BECAUSE_COMMITTED.
- The subchannel tracer receives that physical stream status as
CANCELLED.
Relevant code:
If the server instead returns DEADLINE_EXCEEDED, the tracer receives DEADLINE_EXCEEDED directly.
Desired outlier-detection behavior
We would like to count both of these as endpoint failures:
- The server returns
DEADLINE_EXCEEDED.
- The client deadline expires while waiting for that endpoint.
We would not like to count:
- Explicit application cancellation.
- Context cancellation unrelated to deadline expiry.
- Cancellation of non-winning retry or hedging attempts.
At present, these locally initiated cases appear indistinguishable to an attempt-level ClientStreamTracer because they can all arrive as CANCELLED.
Question
Is there a supported way for a load-balancer policy or ClientStreamTracer to distinguish client deadline expiry from other locally initiated cancellations?
If not, would grpc-java consider exposing the cancellation origin, or otherwise preserving enough information for an outlier-detection policy to make this distinction?
I noticed that gRFC proposal #558 currently proposes excluding locally initiated deadline expiry from outlier-detection counting. Is that intended to be a fixed semantic, or could policies support counting deadline expiry when it is considered a useful endpoint-health signal?
When a client call deadline expires before the server responds, ClientCall.Listener receives
DEADLINE_EXCEEDED, but the subchannel's ClientStreamTracer receivesCANCELLED.The path appears to be:
DEADLINE_EXCEEDED).CANCELLED_BECAUSE_COMMITTED.CANCELLED.Relevant code:
If the server instead returns
DEADLINE_EXCEEDED, the tracer receivesDEADLINE_EXCEEDEDdirectly.Desired outlier-detection behavior
We would like to count both of these as endpoint failures:
DEADLINE_EXCEEDED.We would not like to count:
At present, these locally initiated cases appear indistinguishable to an attempt-level ClientStreamTracer because they can all arrive as
CANCELLED.Question
Is there a supported way for a load-balancer policy or ClientStreamTracer to distinguish client deadline expiry from other locally initiated cancellations?
If not, would grpc-java consider exposing the cancellation origin, or otherwise preserving enough information for an outlier-detection policy to make this distinction?
I noticed that gRFC proposal #558 currently proposes excluding locally initiated deadline expiry from outlier-detection counting. Is that intended to be a fixed semantic, or could policies support counting deadline expiry when it is considered a useful endpoint-health signal?