feat(otlp): add provider_kind and EKS Fargate static tags by lucastemb · Pull Request #2255 · DataDog/saluki

lucastemb · 2026-08-03T19:49:29Z

Summary

Added provider_kind support and EKS Fargate related static tags as part of the ongoing OTLP Ingest metrics pipeline related work.

We have explicitly chosen to omit some Cluster Agent related tags (now tracked by #2254) due to ADP's current narrow implementation and non-blocking nature to our current work

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

How did you test this PR?

Unit tests.

References

Closes: Add provider_kind and EKS Fargate static tags to OTLP metrics #2066

datadog-datadog-us1-prod

The new static-tag path overwrites otlp_config.metrics.tags instead of preserving configured metric tags. Any deployment that combines existing OTLP metric tags with provider_kind or EKS Fargate tagging will silently lose those existing tags, changing metric dimensions and potentially breaking dashboards and monitors.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 6e25d0d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-us1-prod · 2026-08-03T19:54:16Z

+        let mut otlp = otlp.clone();
+        let static_tags = resolve_static_metric_tags(&otlp, &shared.tags, |key| std::env::var(key).ok());
+        if !static_tags.is_empty() {
+            otlp.metrics.tags = static_tags.join(",");


Static tags discard configured OTLP metric tags

Existing OTLP metric dimensions silently disappear for users adopting provider-kind or EKS Fargate tagging, causing incorrect grouping and broken dashboards or monitors.

Assertion details

Input: Configure otlp_config.metrics.tags as service:checkout and enable the new provider_kind:eks setting. resolve_static_metric_tags returns provider_kind:eks, after which from_configuration assigns that result to otlp.metrics.tags.

Expected: Configured metric tags remain present and the new static tags are added, producing service:checkout and provider_kind:eks.

Actual: The assignment at line 129 replaces the entire configured tag string with provider_kind:eks; service:checkout is absent before the metric translator parses the tags.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

Suggested change

otlp.metrics.tags = static_tags.join(",");

otlp.metrics.tags = if otlp.metrics.tags.is_empty() {

static_tags.join(",")

} else {

format!("{},{}", otlp.metrics.tags, static_tags.join(","))

};

This is addressed in the comment below. The Codex bot is effectively pointing out the same thing.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e25d0d689

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-03T19:56:07Z

+        let static_tags = resolve_static_metric_tags(&otlp, &shared.tags, |key| std::env::var(key).ok());
+        if !static_tags.is_empty() {
+            otlp.metrics.tags = static_tags.join(",");


Preserve explicitly configured OTLP metric tags

When provider_kind is non-empty or the process is detected as ECS/EKS Fargate, static_tags is non-empty and this assignment discards the existing otlp_config.metrics.tags value. Since build() later parses only self.otlp.metrics.tags, every explicitly configured OTLP tag silently disappears in those environments; merge or append the static tags instead of replacing the configured string.

Useful? React with 👍 / 👎.

That is intentional parity as seen here. We're correctly overwriting otlp_config.metrics.tags when resolved static tags are non-empty.

chatgpt-codex-connector · 2026-08-03T19:56:07Z

+    /// Static provider classification added to every OTLP metric when non-empty.
+    pub provider_kind: String,
+
+    /// Whether the Agent runs as an EKS Fargate sidecar.
+    pub eks_fargate: bool,


Document defaults and boundaries for the new config fields

The newly added Metrics fields only summarize their purpose, and the added GlobalTags fields similarly only name their source; none explicitly states the empty/false default, relevant boundary behavior, or who should change the setting. Expand the Rustdoc for every added configuration field to cover those required details.

AGENTS.md reference: AGENTS.md:L149-L154

Useful? React with 👍 / 👎.

Fixed in 0b443fa

chatgpt-codex-connector · 2026-08-03T19:56:07Z

    tags.into_shared()
 }

+/// Resolves static tags that the Core Agent adds to OTLP metrics.


Keep the generic OTLP component vendor-neutral

This new Rustdoc names the Core Agent inside lib/saluki-components, which is a generic Saluki crate whose documentation and comments must avoid Datadog Agent-specific names and topology. Describe these as static tags supplied by the server process or configuration instead.

AGENTS.md reference: AGENTS.md:L127-L136

Useful? React with 👍 / 👎.

Fixed in 0b443fa

pr-commenter · 2026-08-03T19:58:03Z

Binary Size Analysis (Agent Data Plane)

Baseline: 9d1d392 · Comparison: e93130c · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.75 MiB (baseline) vs 41.64 MiB (comparison)
Size Change: -119.16 KiB (-0.28%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`figment`	-134.33 KiB	173
`&mut serde_json`	+27.47 KiB	32
`core`	+26.66 KiB	1756
`anon.5ffef35c7768023811ac5ab5fd4f00ff.904.llvm.1595573446908047995`	+17.88 KiB	1
`anon.36a7d0d314b9701965ea5b08796fad33.1185.llvm.15807495580778693563`	-17.80 KiB	1
`anon.032353f61547200fde876e37cf3f8a0b.1182.llvm.1488955665037405644`	+17.80 KiB	1
`serde`	-17.65 KiB	36
`anon.5ffef35c7768023811ac5ab5fd4f00ff.885.llvm.15862285824570942458`	-17.45 KiB	1
`anon.af90fe4baeb6e6b8384b677957b04820.462.llvm.13172964232983855677`	+12.37 KiB	1
`anon.af90fe4baeb6e6b8384b677957b04820.462.llvm.11835343198361071145`	-12.37 KiB	1
`[sections]`	-11.34 KiB	9
`std`	+11.09 KiB	94
`tokio`	+11.07 KiB	795
`saluki_common::task::instrument`	-10.51 KiB	27
`saluki_components::common::datadog`	+10.51 KiB	93
`anon.36a7d0d314b9701965ea5b08796fad33.527.llvm.15807495580778693563`	-9.63 KiB	1
`anon.032353f61547200fde876e37cf3f8a0b.524.llvm.1488955665037405644`	+9.62 KiB	1
`serde_json`	-9.41 KiB	93
`anon.36a7d0d314b9701965ea5b08796fad33.1069.llvm.15807495580778693563`	-9.07 KiB	1
`anon.032353f61547200fde876e37cf3f8a0b.1066.llvm.1488955665037405644`	+9.06 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
 +13e2% +26.0Ki +14e2% +26.0Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::hb214cb615a40f66d
  [NEW] +19.9Ki  [NEW] +19.7Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::h0631d7db0dec70b3
  [NEW] +19.8Ki  [NEW] +19.7Ki    agent_data_plane::internal::env::workload::build_collector::_{{closure}}::hfecfe48366f575fc
  [NEW] +19.6Ki  [NEW] +19.4Ki    saluki_components::transforms::apm_stats::ApmStats::process_trace::hce992ca7b6928d1d
  [NEW] +17.9Ki  [NEW] +17.8Ki    anon.5ffef35c7768023811ac5ab5fd4f00ff.904.llvm.1595573446908047995
  [NEW] +17.8Ki  [NEW]     +76    anon.032353f61547200fde876e37cf3f8a0b.1182.llvm.1488955665037405644
  [NEW] +17.0Ki  [NEW] +16.5Ki    core::ptr::drop_in_place<datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::h5ac45f4756556f26
   +83% +15.4Ki   +84% +15.4Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::h17e28c515cd1a8f5
  [NEW] +14.8Ki  [NEW] +14.7Ki    datadog_agent_config::generated::witness::drive::hf1d766b0da57b759
  [NEW] +13.2Ki  [NEW] +13.0Ki    _<core::marker::PhantomData<T> as serde_core::de::DeserializeSeed>::deserialize::h48f5d2fbe6ebb01f
  [DEL] -14.0Ki  [DEL] -13.9Ki    agent_data_plane::dogstatsd_contexts::artifact::decode_records::h8e2f07551fb41d56
  [DEL] -14.2Ki  [DEL] -14.1Ki    datadog_agent_config::generated::witness::drive::h7d1355f2e03f9c0b
  [DEL] -14.5Ki  [DEL] -14.4Ki    agent_data_plane::dogstatsd_contexts::artifact::decode_records::h4dab89d10acfd1a2
  [DEL] -15.0Ki  [DEL] -14.9Ki    _<figment::value::magic::Tagged<T> as figment::value::magic::Magic>::deserialize_from::hf697faa584ef7446
  [DEL] -15.1Ki  [DEL] -14.9Ki    _<figment::value::magic::RelativePathBuf as figment::value::magic::Magic>::deserialize_from::h18cc13ebe25523a9
  [DEL] -15.8Ki  [DEL] -15.3Ki    core::ptr::drop_in_place<datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::h1fc411b0a78628dd
  [DEL] -16.2Ki  [DEL] -16.0Ki    saluki_components::transforms::apm_stats::span_concentrator::SpanConcentrator::flush::ha184bfce3e1498d8
  [DEL] -17.4Ki  [DEL] -17.4Ki    anon.5ffef35c7768023811ac5ab5fd4f00ff.885.llvm.15862285824570942458
  [DEL] -17.8Ki  [DEL]     -76    anon.36a7d0d314b9701965ea5b08796fad33.1185.llvm.15807495580778693563
  [DEL] -32.5Ki  [DEL] -32.4Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::h273918a7d610bdfa
  -1.3%  -127Ki  -1.2% -93.1Ki    [14202 Others]
  -0.3%  -119Ki  -0.2% -83.9Ki    TOTAL

pr-commenter · 2026-08-03T20:12:39Z

Regression Detector (Agent Data Plane)

Run ID: 14b87727-d590-494b-b860-ec97fb78c02e
Baseline: 9d1d3921 · Comparison: e93130ca · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_low	memory	⚪ +0.46	metrics profiles logs
quality_gates_rss_idle	memory	⚪ +0.37	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.15	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.11	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.23	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 144 MiB ≤ 145 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 49.4 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 76.7 MiB ≤ 80 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 203 MiB ≤ 205 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 32.1 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

tobz · 2026-08-03T23:50:00Z

+    if !otlp.metrics.kubernetes_kubelet_nodename.is_empty() {
+        tags.push(format!("eks_fargate_node:{}", otlp.metrics.kubernetes_kubelet_nodename));
+    } else {
+        warn!("Couldn't build the 'eks_fargate_node' tag: kubernetes_kubelet_nodename is not configured.");


Is this log message (and the equivalent one below) meant to explicitly match a log message from the Core Agent?

It might be good to try and reword it to make it more friendly to users, in a "problem-solution" pattern:

Tag 'eks_fargate_node' will be missing from telemetry ingested via OTLP due to missing configuration data. Ensure 'kubernetes_kubelet_nodename' is set in the configuration.

If this is just something they can set (but more on that in a second), then we should explicitly be telling them not only what to set but where to set it. As is, we're not being that explicit that this is a configuration setting.

Separately: is this actually something users are expected to set manually, or something that would be configured by the Helm chart or Operator? If it's the latter, then we actually might not want to have the "solution" part if there's no way for them to act on it.

Seems like it's something that users set manually. I do think the problem-solution pattern for the error message is better than what I had and was added in c299b79 😄

webern · 2026-08-04T09:07:19Z

+    let ecs_fargate =
+        environment("AWS_EXECUTION_ENV").as_deref() == Some("AWS_ECS_FARGATE") || environment("ECS_FARGATE").is_some();


This is bothering me. It's an environmental side-effect buried in an otherwise relatively pure function. I feel like this check belongs somewhere that is dedicated to setup (right?) and that its environment determination should be passed in to this function. I'm not sure if there's a "place" for that, but it jumped out at me as a kind of hidden form of config/environment sensitivity.

You're right. Fixed in c299b79. I made it such that the check for ECS_Fargate occurs much earlier and is propagated through instead of belonging to that hacky conditional statement you called out.

… message

feat(otlp): add provider_kind and EKS Fargate static tags

6e25d0d

dd-octo-sts Bot added area/components Sources, transforms, and destinations. area/docs Reference documentation. source/otlp OTLP source. labels Aug 3, 2026

lucastemb marked this pull request as ready for review August 3, 2026 19:51

lucastemb requested a review from a team as a code owner August 3, 2026 19:51

datadog-datadog-us1-prod Bot reviewed Aug 3, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

fix(otlp): restore dogstatsd shared ownership of provider_kind

fd9596a

refactor(otlp): improve docs for new config fields and improve testing

0b443fa

tobz reviewed Aug 3, 2026

View reviewed changes

webern reviewed Aug 4, 2026

View reviewed changes

refactor(otlp): global fargate check and added more descriptive error…

c299b79

… message

dd-octo-sts Bot added the area/config Configuration. label Aug 5, 2026

Merge branch 'main' into lt/2066

e93130c

		let ecs_fargate =
		environment("AWS_EXECUTION_ENV").as_deref() == Some("AWS_ECS_FARGATE") \|\| environment("ECS_FARGATE").is_some();

Uh oh!

Conversation

lucastemb commented Aug 3, 2026

Summary

Change Type

How did you test this PR?

References

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-us1-prod Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pr-commenter Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

pr-commenter Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pr-commenter Bot commented Aug 3, 2026 •

edited

Loading

pr-commenter Bot commented Aug 3, 2026 •

edited

Loading