Skip to content

fix(opentelemetry source): implement standard gRPC health checking protocol - #26025

Open
stigglor wants to merge 2 commits into
vectordotdev:masterfrom
stigglor:opentelemetry-grpc-hc
Open

fix(opentelemetry source): implement standard gRPC health checking protocol#26025
stigglor wants to merge 2 commits into
vectordotdev:masterfrom
stigglor:opentelemetry-grpc-hc

Conversation

@stigglor

@stigglor stigglor commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

A gRPC health check against the opentelemetry source's gRPC listener hits an unrouted
method and gets back a bare HTTP 404 with no grpc-status, so anything speaking the gRPC
health checking protocol has nothing to evaluate. An ALB target group or a Kubernetes
probe has to check an Export method instead and accept a range of statuses wide enough
to cover the decoder rejecting an empty body, which rides on an error path rather than a
health signal. Every such probe also logs Grpc error. error=Received 404 Not Found at
ERROR level against the source.

The vector source already registers grpc.health.v1.Health via tonic_health (#24916)
and the API replaced its custom Health RPC with the same service (#25139), so this applies
the established pattern to the source that missed out. The reporter serves the aggregate
(empty) service name, which is what a probe sends unless told otherwise, and each of the
three OTLP services is reported by name so a probe can target a single signal. Those names
come from the generated NamedService implementations rather than string literals so they
cannot drift from the routes actually registered.

Status is static SERVING, matching the vector source: the reporter handle is dropped
after registration, so the source does not flip to NOT_SERVING while draining. That
seems worth revisiting for both sources but is out of scope here. The HTTP listener on
4318 is untouched.

Vector configuration

sources:
  otlp:
    type: opentelemetry
    grpc:
      address: 127.0.0.1:14317
    http:
      address: 127.0.0.1:14318
    use_otlp_decoding: true

sinks:
  drop:
    type: blackhole
    inputs:
      - otlp.logs
      - otlp.metrics
      - otlp.traces

How did you test this PR?

Added standard_grpc_health_check_works, modelled on the vector source test of the same
name. It asserts SERVING for the aggregate and the three OTLP service names, and
NOT_FOUND for an unregistered one.

cargo nextest run -p vector --no-default-features --features sources-opentelemetry -E 'test(sources::opentelemetry)'
# 25 passed

I also ran the config above against a locally built binary and probed it over raw HTTP/2
with curl, since it is the wire behaviour that matters to a load balancer:

Request on 14317 Result
Health/Check, empty service HTTP/2 200, content-type: application/grpc, trailer grpc-status: 0, body 08 01 (SERVING)
Health/Check, each OTLP service name grpc-status: 0, SERVING
Health/Check, not.a.real.Service grpc-status: 5, service not registered
TraceService/Export, empty message grpc-status: 3, Invalid OTLP data: expected one of {Traces}, unchanged
does.not.Exist/Method bare HTTP/2 404, no grpc-status, which is what health checks used to get

grpc_health_probe is the practical follow-on for Kubernetes probes; I did not have it
installed, hence the raw HTTP/2 probes above.

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

…otocol

A health check against the OTLP gRPC listener hit an unrouted method and got back a
bare HTTP 404 with no grpc-status, so anything that speaks gRPC health checking had
nothing to evaluate. In practice that means an ALB target group or a Kubernetes probe
has to check an Export method instead and accept a range of statuses wide enough to
cover the decoder rejecting an empty body, which is a workaround riding on an error
path rather than a health signal. Every such probe also logged a Grpc error at ERROR
level against the source.

The vector source already registers grpc.health.v1.Health via tonic_health (vectordotdev#24916),
and the API replaced its custom Health RPC with the same service (vectordotdev#25139), so this
applies the established pattern to the source that missed out. The reporter serves the
aggregate (empty) service name, which is what a probe sends by default, and each of the
three OTLP services is reported by name so a probe can target a single signal. Those
names come from the generated NamedService impls so they cannot drift from the
registered routes.

Health checks now answer SERVING on the OTLP gRPC port, an unregistered name gets
NOT_FOUND rather than a 404, and grpc-health-probe works against port 4317.

Status is static SERVING, matching the vector source: the reporter handle is dropped
after registration, so the source does not flip to NOT_SERVING while draining. Worth
revisiting for both sources, but out of scope here. The HTTP listener is untouched.
@stigglor
stigglor requested a review from a team as a code owner August 5, 2026 05:44
@github-actions github-actions Bot added the domain: sources Anything related to the Vector's sources label Aug 5, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2734cab6b9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/opentelemetry/config.rs
Registering the health service on a source's gRPC listener put probes through
DecompressionAndMetricsLayer, which emits BytesReceived for every successful request, so each
check added a few bytes to that source's component_received_bytes_total. A probe carries no
events, so an installation probing every few seconds saw the metric drift upward on traffic it
never accepted.

The layer wraps the whole server and cannot be applied per route, so it now skips byte emission
when the request path belongs to grpc.health.v1.Health. Probes still go through decompression,
because a client is free to compress one. The gRPC server request and duration metrics continue
to count probes, since those describe server activity rather than received data.

This applies to the vector source as well, which has registered the health service since vectordotdev#24916.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: sources Anything related to the Vector's sources

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant