feat(metrics-v3): add Datadog V3 payload encoder#2223
Conversation
📚 Documentation Check Results📦
|
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: a743570 | Docs | Datadog PR Page | Give us feedback! |
🔒 Cargo Deny Results✅ No issues found! 📦
|
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
atanzu
force-pushed
the
mark.kirichenko/AGTMETRICS-536/add-metrics-v3-encoder
branch
from
db0bdea to
f0b893c
Compare
BenchmarksComparisonBenchmark execution time: 2026-07-10 15:15:23 Comparing candidate commit a743570 in PR branch Found 3 performance improvements and 4 performance regressions! Performance is the same for 170 metrics, 10 unstable metrics.
|
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
atanzu
force-pushed
the
mark.kirichenko/AGTMETRICS-536/add-metrics-v3-encoder
branch
from
f0b893c to
f6a9fc0
Compare
Allows to encode metric payloads using V3 columnar format. This library enables users to encode their metric payloads using an efficient column-based protocol. For performance and compatibility reasons (we also want to keep this crate `no_std` so we could use it in very resource-constrained environments) this crate does manual protobuf serialization. Signed-off-by: Mark Kirichenko <mark.kirichenko@datadoghq.com>
atanzu
force-pushed
the
mark.kirichenko/AGTMETRICS-536/add-metrics-v3-encoder
branch
from
f6a9fc0 to
a743570
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a743570c4b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if no_index { | ||
| self.writer.types[self.metric_idx] |= FLAG_NO_INDEX; | ||
| } |
There was a problem hiding this comment.
If set_origin is called more than once for the same metric, a later call with no_index == false does not undo an earlier no_index == true; the bit is only ever ORed into types. In that scenario the finalized payload still marks the series as flagNoIndex, so the metric can be hidden/non-indexed even though the latest setter value requested indexing.
Useful? React with 👍 / 👎.
| self.writer.sketch_num_bins.push(bin_keys.len() as u64); | ||
|
|
||
| let key_start = self.writer.sketch_bin_keys.len(); | ||
| self.writer.sketch_bin_keys.extend_from_slice(bin_keys); | ||
| self.writer.sketch_bin_cnts.extend_from_slice(bin_counts); |
There was a problem hiding this comment.
When bin_keys.len() != bin_counts.len(), this records sketch_num_bins from the key count but writes a different number of count entries. A decoder will consume that advertised number of counts for the sketch, so short count slices steal counts from the next sketch and long slices leave extras that shift later sketches, corrupting distribution payloads instead of rejecting the invalid input.
Useful? React with 👍 / 👎.
| /// | ||
| /// Use the setter methods to configure the metric, add points with [`add_point`](Self::add_point), | ||
| /// then call [`close`](Self::close) to finalize. | ||
| pub struct V3MetricBuilder<'a> { |
There was a problem hiding this comment.
If a caller lets V3MetricBuilder fall out of scope without calling close, the metric keeps its default zero value type while its point values remain in vals_float64; finalizing the writer then encodes those non-zero points as zeros and leaves stray value-column entries. Because close has no return value and the builder is not enforced by the type system, this is an easy way for normal API use to silently corrupt a payload.
Useful? React with 👍 / 👎.
| pub fn add_point(&mut self, timestamp: i64, value: f64) { | ||
| self.writer.timestamps.push(timestamp); | ||
| self.writer.vals_float64.push(value); | ||
| self.writer.num_points[self.metric_idx] += 1; |
There was a problem hiding this comment.
When the builder was created with V3MetricType::Sketch, calling the generic add_point path records only one timestamp/value and no sketch count/bin metadata, but the encoded type still tells decoders to read three summary values plus a count and sketch bins for each point. That malformed sketch consumes subsequent value columns incorrectly (or fails to decode), so this method should reject sketch metrics rather than accepting them silently.
Useful? React with 👍 / 👎.
| self.writer.timestamps.push(timestamp); | ||
|
|
||
| // Count goes in sint64, sum/min/max go in float64 | ||
| self.writer.vals_sint64.push(count); | ||
| self.writer.vals_float64.push(sum); | ||
| self.writer.vals_float64.push(min); | ||
| self.writer.vals_float64.push(max); |
There was a problem hiding this comment.
When the builder was created for a Count/Rate/Gauge, add_sketch still appends sketch summaries, counts, and bin columns, but the metric type remains non-sketch so decoders will read only one normal value for the point. The extra values and sketch columns then shift the cursors used for following metrics, corrupting the rest of the payload rather than rejecting the mismatched API call.
Useful? React with 👍 / 👎.
1 participant
What does this PR do?
Add a new library which allows to encode metric payloads using V3 columnar format.
Motivation
This library enables users to encode their metric payloads using an efficient column-based protocol. For performance and compatibility reasons (we also want to keep this crate
no_stdso we could use it in very resource-constrained environments) this crate does manual protobuf serialization.Additional Notes
How to test the change?
We do correctness test by comparing the resulting encoded payload with the one produced from protobuf-generated code.