feat(otlp): added support for datadog_metric.as_type=rate for sums

[lucastemb]

Summary

When a metric is emitted with the attribute as_type=rate, we should strive to emit delta Sums as a Rate.

as_type=rate will be treated as case insensitive

Additional plumbing is added to make additional metric types such as Count and Gauge no-ops while making unknown types emit an error.

Lastly, after investigating #2076, we discovered that with_delta_interval sits behind a feature gate with the default false with no way to toggle it to true, meaning, that the duration will stay zero since we have no way to compute the interval.

Consequently, we will emit a warning the first time that a metric is observed notifying no delta interval is available.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

Unit tests.

The issue calls for correctness tests, but it should be deferred and opened as a subsequent issue once we do more work around interval-inference in the future.

References

• Closes: Support datadog.metric.as_type=rate for OTLP delta sums #2053

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 9d1d392 · Comparison: b5ec63e · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.75 MiB (baseline) vs 41.62 MiB (comparison)
Size Change: -133.16 KiB (-0.31%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
figment	-146.56 KiB	186
&mut serde_json	+26.38 KiB	34
serde	-22.57 KiB	37
saluki_components::transforms::dogstatsd_mapper	+21.93 KiB	9
serde_with	-18.66 KiB	23
saluki_components::common::datadog	+18.42 KiB	97
anon.36a7d0d314b9701965ea5b08796fad33.1185.llvm.12982751287224477828	+17.80 KiB	1
anon.36a7d0d314b9701965ea5b08796fad33.1185.llvm.15807495580778693563	-17.80 KiB	1
prost	-14.32 KiB	133
std	+13.72 KiB	61
[sections]	-10.49 KiB	8
rmp	-9.99 KiB	24
anon.36a7d0d314b9701965ea5b08796fad33.527.llvm.12982751287224477828	+9.63 KiB	1
anon.36a7d0d314b9701965ea5b08796fad33.527.llvm.15807495580778693563	-9.63 KiB	1
anon.36a7d0d314b9701965ea5b08796fad33.1069.llvm.12982751287224477828	+9.07 KiB	1
anon.36a7d0d314b9701965ea5b08796fad33.1069.llvm.15807495580778693563	-9.07 KiB	1
tracing	-9.06 KiB	20
&mut rmp_serde	+9.01 KiB	8
otlp_protos::otlp_include::opentelemetry	+8.88 KiB	137
anon.36a7d0d314b9701965ea5b08796fad33.16.llvm.12982751287224477828	+8.84 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +32.4Ki  [NEW] +32.3Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::haa8671f97e5edf15
  [NEW] +17.8Ki  [NEW]     +76    anon.36a7d0d314b9701965ea5b08796fad33.1185.llvm.12982751287224477828
  [NEW] +14.7Ki  [NEW] +14.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::map_to_dd_format::h2e89fdb9a6b2ab11
  [NEW] +13.1Ki  [NEW] +13.0Ki    _<core::marker::PhantomData<T> as serde_core::de::DeserializeSeed>::deserialize::h48f5d2fbe6ebb01f
  [NEW] +13.0Ki  [NEW] +12.8Ki    saluki_components::common::datadog::proxy::_::_<impl serde_core::de::Deserialize for saluki_components::common::datadog::proxy::ProxyConfiguration>::deserialize::hc8778832a740dfe8
  [NEW] +12.9Ki  [NEW] +12.8Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h66a92955291a3b92
  +585% +10.6Ki  +650% +10.6Ki    _<saluki_components::sources::otlp::logs::translator::OtlpLogsTranslator as core::iter::traits::iterator::Iterator>::next::h5a4d63054fce89e2
  [NEW] +9.66Ki  [NEW] +9.50Ki    _<&mut serde_json::de::Deserializer<R> as serde_core::de::Deserializer>::deserialize_struct::hdb61da5fef4e4539
  [NEW] +9.64Ki  [NEW] +9.50Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::map_histogram_metrics::h5362ad340b489630
  [NEW] +9.63Ki  [NEW]     +79    anon.36a7d0d314b9701965ea5b08796fad33.527.llvm.12982751287224477828
  [DEL] -9.63Ki  [DEL]     -79    anon.36a7d0d314b9701965ea5b08796fad33.527.llvm.15807495580778693563
  [DEL] -9.93Ki  [DEL] -9.79Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::map_histogram_metrics::hecee1048a6006e3f
  [DEL] -10.8Ki  [DEL] -10.7Ki    saluki_components::sources::otlp::logs::transform::transform_log_record::h2af58f29efba97f7
  [DEL] -12.8Ki  [DEL] -12.6Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::hdc5c7a358bf201fd
  [DEL] -12.9Ki  [DEL] -12.8Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hadaeef09f71d33b1
  [DEL] -15.0Ki  [DEL] -14.9Ki    _<figment::value::magic::Tagged<T> as figment::value::magic::Magic>::deserialize_from::hf697faa584ef7446
  [DEL] -15.1Ki  [DEL] -14.9Ki    _<figment::value::magic::RelativePathBuf as figment::value::magic::Magic>::deserialize_from::h18cc13ebe25523a9
  [DEL] -15.4Ki  [DEL] -15.3Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::map_to_dd_format::h44f06bece734e31b
  [DEL] -17.8Ki  [DEL]     -76    anon.36a7d0d314b9701965ea5b08796fad33.1185.llvm.15807495580778693563
  [DEL] -42.4Ki  [DEL] -42.3Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h07d37085d1c426ac
  -1.6%  -114Ki  -1.8%  -101Ki    [6723 Others]
  -0.3%  -133Ki  -0.3%  -119Ki    TOTAL

[datadog-datadog-prod-us1]

The new as_type=rate path constructs every converted delta Sum with a zero interval. Both Datadog encoders divide the value by that interval and discard the resulting non-finite value, so a valid delta Sum such as 42 is sent as a rate series with no points and is silently lost.

_{📊 Validated against 1 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 94add5d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[datadog-datadog-prod-us1]

Zero rate interval drops converted delta Sums

All default-config delta Sums requesting as_type=rate are silently omitted from Datadog metric payloads, causing users to lose those measurements.

Assertion details

• Input: An OTLP delta Sum with datadog.metric.as_type=rate and value 42, when interval inference is unavailable (the default configuration).
• Expected: The converted metric should preserve the value in an encodable rate representation, or use a defined fallback that does not discard the point.
• Actual: The diff creates MetricValues::Rate(..., Duration::ZERO). The v1 and v2 encoders divide the value by interval.as_secs_f64(), producing infinity, and their non-finite filter removes the point; the payload retains rate type and interval 0 but has no points.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[lucastemb]

Same rationale as below.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 94add5d755

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid zero-interval rate metrics

For any OTLP delta Sum datapoint with datadog.metric.as_type=rate, this creates a MetricValues::Rate with Duration::ZERO; I checked the Datadog metrics encoders and they divide rate values by the stored interval before filtering finite points, so every finite value becomes inf/NaN and is dropped from the payload instead of being emitted as a rate. Use the inferred delta interval when available, and a non-zero/no-scale representation for the no-interval case.

Useful? React with 👍 / 👎.

[lucastemb]

Good catch. Fixed in 5636c9e

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: d45bdcef-3f83-4e05-ae98-55be4accb02f
Baseline: 9d1d3921 · Comparison: b5ec63e3 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_low	memory	⚪ +0.65	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.16	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.09	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.15	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.23	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 141 MiB ≤ 145 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 49.3 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 76.1 MiB ≤ 80 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 202 MiB ≤ 205 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 32 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[webern]

This one went stale. I'm having a little bit of trouble understanding if it does the "right thing" with duration=0 "rate" data points. Is the right thing to send them to the backend? Or are we supposed to drop them?

One thing that bugs me a little bit too is code duplication for v1, v2, v3. Do we have a precedent for sharing code that all encoding versions need to do? Also, is there any risk inherent in the fact that the encoders are not source-aware? In other words, are we likely to encounter cases where OTLP encoding rules could differ from DogStatsD? (These are questions for @tobz, I suppose)

[lucastemb]

This one went stale. I'm having a little bit of trouble understanding if it does the "right thing" with duration=0 "rate" data points. Is the right thing to send them to the backend? Or are we supposed to drop them?

One thing that bugs me a little bit too is code duplication for v1, v2, v3. Do we have a precedent for sharing code that all encoding versions need to do? Also, is there any risk inherent in the fact that the encoders are not source-aware? In other words, are we likely to encounter cases where OTLP encoding rules could differ from DogStatsD? (These are questions for @tobz, I suppose)

@webern Yes, as far as I can tell, we are doing the "right thing" with rate and zero. That's how the Core Agent handles it. It forwards the metric to standard intake as an unnormalized rate (so it just doesn't divide by zero in our case). I think the main thing is what you raise with your last questions because consolidating that could affect DogStatsD behavior in theory. So I should raise it with @tobz and get a second opinion.