From e7292ab5e1f154c8d3776e450fc02167ce6fa03f Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Tue, 18 Aug 2026 15:13:34 -0500 Subject: [PATCH 1/3] docs(observability): document metadata promotion Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- .../observability/opentelemetry.mdx | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index e9af34819..2bf8d0b0a 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -91,6 +91,7 @@ type = "gen_ai" endpoint = "http://localhost:4318/v1/traces" transport = "http_binary" service_name = "agent-service" +promote_metadata_prefixes = ["nv."] max_queue_size = 4096 max_export_batch_size = 512 scheduled_delay_millis = 1000 @@ -143,6 +144,34 @@ to the other exporters. | `mark_projection` | `inherit` | Mark representation for `full` and `openinference`: `inherit`, `event`, or `tool`. | | `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `full` and `openinference` projection. | | `attribute_mappings` | `[]` | `{ key, alias }` copies applied by `full` and `openinference` projection. | +| `promote_metadata_prefixes` | `[]` | Literal Event metadata prefixes copied to top-level span attributes. | + +## Event Metadata Promotion + +Set `promote_metadata_prefixes` on a trace endpoint to copy selected keys from +the final sanitized Event metadata into that endpoint's OpenTelemetry output. +Prefixes use the Event metadata key syntax: ASCII letters, numbers, underscores, +and hyphens in segments separated by single dots. A trailing dot is optional. +For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes. +Leading or repeated dots, whitespace, other punctuation, and glob expressions +such as `"nv.*"` are rejected. + +Scope metadata is promoted when the scope-end Event completes the span, so each +promoted key has one final value. Mark metadata is promoted to the attributes +of the projected span event or tool span. The `gen_ai` projection continues to +omit Marks. + +Promotion supports strings, booleans, signed 64-bit integers, floating-point +numbers, and homogeneous arrays of those primitive types. Nulls, empty arrays, +nested arrays, mixed-type arrays, objects, and unsigned integers larger than +the signed 64-bit range are omitted. Relay records a bounded runtime diagnostic +containing the key and reason, but it does not record the rejected value or stop +trace export. + +Projection-owned attributes and configured attribute-mapping aliases take +precedence over promoted metadata with the same key. Promotion does not modify +the Event or ATOF payload. Use `resource_attributes` instead for static values +that must be attached to every span from an endpoint. ## Log and Metric Endpoint Resolution From 2901e8e1d7b6b6dfe9ac6a4c90db505aa1857295 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Tue, 18 Aug 2026 18:22:49 -0500 Subject: [PATCH 2/3] docs(observability): clarify literal metadata prefixes Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- .../observability/opentelemetry.mdx | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index 2bf8d0b0a..2144e3b4b 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -156,6 +156,12 @@ For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes. Leading or repeated dots, whitespace, other punctuation, and glob expressions such as `"nv.*"` are rejected. +Matching is case-sensitive and compares the beginning of each key literally. +Relay does not infer a dot or metadata-key segment boundary. For example, +`"nv."` selects keys that begin with `nv.`, `"nv_"` selects keys that begin +with `nv_`, and `"user"` selects keys such as `username` and `user_api_key`. +Configure the narrowest prefix that selects the metadata you intend to export. + Scope metadata is promoted when the scope-end Event completes the span, so each promoted key has one final value. Mark metadata is promoted to the attributes of the projected span event or tool span. The `gen_ai` projection continues to @@ -169,9 +175,15 @@ containing the key and reason, but it does not record the rejected value or stop trace export. Projection-owned attributes and configured attribute-mapping aliases take -precedence over promoted metadata with the same key. Promotion does not modify -the Event or ATOF payload. Use `resource_attributes` instead for static values -that must be attached to every span from an endpoint. +precedence over promoted metadata with the same key. Relay also omits selected +keys in namespaces owned by Relay or supported semantic projections: +`nemo_relay.`, `gen_ai.`, `error.`, `exception.`, `input.`, `output.`, `llm.`, +`openinference.`, `server.`, `service.`, `session.`, `tool.`, `tool_call.`, and +`user.`. Relay omits the bare `metadata` key as well. Rejected keys produce a +rate-limited operational diagnostic without dropping the Event or span. + +Promotion does not modify the Event or ATOF payload. Use `resource_attributes` +instead for static values that must be attached to every span from an endpoint. ## Log and Metric Endpoint Resolution From 136c5bb2a2b5425ee5c5fa1aa27f69abb4309f80 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Tue, 18 Aug 2026 23:03:11 -0500 Subject: [PATCH 3/3] docs(observability): align metadata promotion contract Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- .../observability/opentelemetry.mdx | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index 2144e3b4b..8cfdf0a54 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -150,8 +150,18 @@ to the other exporters. Set `promote_metadata_prefixes` on a trace endpoint to copy selected keys from the final sanitized Event metadata into that endpoint's OpenTelemetry output. -Prefixes use the Event metadata key syntax: ASCII letters, numbers, underscores, -and hyphens in segments separated by single dots. A trailing dot is optional. +The injection and promotion stages use the following contract: + +| Behavior | Contract | +|---|---| +| Exact metadata keys | ASCII letters, numbers, underscores, and hyphens in nonempty segments separated by single dots. | +| Promotion prefixes | The same key syntax with an optional trailing dot. Matching is literal and case-sensitive. | +| Values | Strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays containing one supported primitive type. | +| Rejected values | Nulls, objects, nested arrays, mixed-type arrays, and unsigned integers larger than `i64::MAX`. | +| Injector conflicts | Existing Event metadata wins. Otherwise, Relay orders injectors by ascending priority and registration name, and the first successful insertion wins. | +| Scope lifecycle | Relay applies injector precedence independently to Scope-start and Scope-end Events. A Scope-end key is authoritative when Relay constructs the final span; when Scope-end omits the key, the Scope-start value remains. | +| OpenTelemetry collisions | Projection-owned attributes and configured attribute-mapping aliases win over promoted metadata. | + For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes. Leading or repeated dots, whitespace, other punctuation, and glob expressions such as `"nv.*"` are rejected. @@ -162,17 +172,17 @@ Relay does not infer a dot or metadata-key segment boundary. For example, with `nv_`, and `"user"` selects keys such as `username` and `user_api_key`. Configure the narrowest prefix that selects the metadata you intend to export. -Scope metadata is promoted when the scope-end Event completes the span, so each -promoted key has one final value. Mark metadata is promoted to the attributes -of the projected span event or tool span. The `gen_ai` projection continues to -omit Marks. +Scope-start and Scope-end are separate Event records, and Relay runs the +ordered injector chain independently for each record. When Scope-end completes +the span, a metadata key present on that Event replaces the corresponding +promoted Scope-start value. Mark metadata is promoted to the attributes of the +projected span event or tool span. The `gen_ai` projection continues to omit +Marks. Promotion supports strings, booleans, signed 64-bit integers, floating-point -numbers, and homogeneous arrays of those primitive types. Nulls, empty arrays, -nested arrays, mixed-type arrays, objects, and unsigned integers larger than -the signed 64-bit range are omitted. Relay records a bounded runtime diagnostic -containing the key and reason, but it does not record the rejected value or stop -trace export. +numbers, empty arrays, and homogeneous arrays of those primitive types. Relay +omits rejected values and records a bounded runtime diagnostic containing the +key and reason, but it does not record the rejected value or stop trace export. Projection-owned attributes and configured attribute-mapping aliases take precedence over promoted metadata with the same key. Relay also omits selected