feat(prometheus_scrape): support auto discovery in k8s by LeeTeng2001 · Pull Request #26013 · vectordotdev/vector

LeeTeng2001 · 2026-08-04T13:56:26Z

Summary

Support discovery of kubernetes pods via annotations

Vector configuration

Since I'm testing this own own my own cluster, I used official helm chart with following values

role: Agent

image:
  repository: git.leeteng.com/leeteng2001/vector # dev repository, this is mine though
  tag: dev
  pullPolicy: Always

customConfig:
  data_dir: /vector-data-dir
  sources:
    demo_metrics:
      type: prometheus_scrape
      scrape_interval_secs: 15
      scrape_timeout_secs: 5
      targets:
        - kubernetes:
            role: pod
            namespaces: [demo]
            label_selector: "app=metrics-demo"
            use_self_node_only: true
  sinks:
    stdout:
      type: console
      inputs: [demo_metrics]
      encoding:
        codec: json

How did you test this PR?

Run a k3s node, deploy sample applications with pod annotations, deploy vector with above helm value configuration

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Closes: Autodiscovery of Prometheus endpoints for scraping #2303

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

github-actions · 2026-08-04T13:56:49Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d1f64e3381

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-04T14:22:44Z

+#[cfg(feature = "kubernetes")]
+pub(crate) mod kubernetes_sd;


Gate Kubernetes SD on prometheus_scrape

When a custom build enables another Prometheus source together with Kubernetes but leaves sources-prometheus-scrape off (for example --no-default-features --features sources-prometheus-remote-write,sources-kubernetes_logs), this #[cfg(feature = "kubernetes")] still pulls in kubernetes_sd, which imports scrape-only items such as PrometheusParseError gated behind sources-prometheus-scrape. That feature set stops compiling even though it does not configure prometheus_scrape; please require both features for this module.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-04T14:22:45Z

+const fn default_delay_deletion_ms() -> u64 {
+    60_000


Stop retaining deleted pods as scrape targets

With the default delay_deletion_ms, a Pod delete event stays in the reflector store for 60 seconds, and collect_targets uses that store on every scrape tick. Because each scrape already clones its Targets before issuing requests, this retention is not needed for in-flight scrapes; instead, deleted pods continue to be scraped for several intervals and can produce errors or, after IP reuse, metrics labeled with the deleted pod's metadata.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-04T14:22:45Z

+        ..Default::default()
+    };
+
+    let api = Api::<Pod>::all(client.clone());


Use namespaced watches for namespace-restricted discovery

When namespaces is set, this still creates an all-namespaces Pods API and relies on field/client-side filters. A Vector service account granted only a Role in the requested namespace will be forbidden from the cluster-scope watch, so namespace-restricted configs fail unless they also have cluster-wide pod list/watch permissions.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-04T14:22:45Z

+                TargetConfig::Static { urls } => {
+                    for s_url in urls {
+                        let uri = s_url.parse::<Uri>().context(sources::UriParseSnafu)?;
+                        static_urls.push(build_url(&uri, &self.query));


Avoid appending static target query parameters twice

When targets contains both a static group and a kubernetes group and the source-level query option is set, these static URLs are stored after build_url already appends the query; then the Kubernetes path converts them to Targets and scrape_target calls build_url(&target.uri, &cfg.query) again. Static targets in mixed configs therefore receive duplicate parameters, unlike static-only configs.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-04T14:22:45Z

+    };
+    Uri::builder()
+        .scheme(scheme.as_str())
+        .authority(format!("{host}:{port}"))


Bracket IPv6 pod IPs before building scrape URIs

In IPv6-only or dual-stack clusters where pod.status.pod_ip is an IPv6 literal, formatting the authority as {host}:{port} produces an invalid URI such as fd00::1:9100. Those annotated pods are skipped with URI build errors instead of being scraped; wrap IPv6 hosts in brackets before setting the authority.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-04T14:22:45Z

+            .buffer_unordered(usize::MAX)
+            .flat_map(stream::iter);
+
+        if out.send_event_stream(&mut events_stream).await.is_err() {


Preserve scrape cadence for slow Kubernetes batches

When any discovered target takes longer than scrape_interval_secs, this awaits the whole finite per-tick scrape stream before polling the next interval. Unlike the existing static path, Kubernetes discovery mode skips overlapping scrapes and lowers the sample frequency for every target in the group whenever one scrape runs long; schedule ticks independently of batch completion.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-04T14:22:45Z

+    };
+
+    let (parts, body) = response.into_parts();
+    let body: Bytes = match http_body::Body::collect(body).await {


Apply the scrape timeout while reading response bodies

When an exporter sends response headers and then stalls the body, the configured timeout has already ended because it only wraps client.send; Body::collect can await forever. In this Kubernetes path the per-tick stream must finish before the loop continues, so one bad pod can freeze all future scrapes for the source.

Useful? React with 👍 / 👎.

github-actions Bot added the domain: sources Anything related to the Vector's sources label Aug 4, 2026

LeeTeng2001 marked this pull request as ready for review August 4, 2026 14:03

LeeTeng2001 requested a review from a team as a code owner August 4, 2026 14:03

LeeTeng2001 and others added 4 commits August 4, 2026 22:03

working implementation

2ca1bae

align with vector api

141a87a

fix only single kubernetes target group is supported

3b9a7c6

fix: add author

7db50fb

LeeTeng2001 force-pushed the dev/prometheus-autoscrape branch from d1f64e3 to 7db50fb Compare August 4, 2026 14:03

chatgpt-codex-connector Bot reviewed Aug 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(prometheus_scrape): support auto discovery in k8s - #26013

feat(prometheus_scrape): support auto discovery in k8s#26013
LeeTeng2001 wants to merge 4 commits into
vectordotdev:masterfrom
LeeTeng2001:dev/prometheus-autoscrape

LeeTeng2001 commented Aug 4, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

LeeTeng2001 commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

LeeTeng2001 commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading