diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index e9af34819..8cfdf0a54 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -91,6 +91,7 @@ type = "gen_ai" endpoint = "http://localhost:4318/v1/traces" transport = "http_binary" service_name = "agent-service" +promote_metadata_prefixes = ["nv."] max_queue_size = 4096 max_export_batch_size = 512 scheduled_delay_millis = 1000 @@ -143,6 +144,56 @@ to the other exporters. | `mark_projection` | `inherit` | Mark representation for `full` and `openinference`: `inherit`, `event`, or `tool`. | | `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `full` and `openinference` projection. | | `attribute_mappings` | `[]` | `{ key, alias }` copies applied by `full` and `openinference` projection. | +| `promote_metadata_prefixes` | `[]` | Literal Event metadata prefixes copied to top-level span attributes. | + +## Event Metadata Promotion + +Set `promote_metadata_prefixes` on a trace endpoint to copy selected keys from +the final sanitized Event metadata into that endpoint's OpenTelemetry output. +The injection and promotion stages use the following contract: + +| Behavior | Contract | +|---|---| +| Exact metadata keys | ASCII letters, numbers, underscores, and hyphens in nonempty segments separated by single dots. | +| Promotion prefixes | The same key syntax with an optional trailing dot. Matching is literal and case-sensitive. | +| Values | Strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays containing one supported primitive type. | +| Rejected values | Nulls, objects, nested arrays, mixed-type arrays, and unsigned integers larger than `i64::MAX`. | +| Injector conflicts | Existing Event metadata wins. Otherwise, Relay orders injectors by ascending priority and registration name, and the first successful insertion wins. | +| Scope lifecycle | Relay applies injector precedence independently to Scope-start and Scope-end Events. A Scope-end key is authoritative when Relay constructs the final span; when Scope-end omits the key, the Scope-start value remains. | +| OpenTelemetry collisions | Projection-owned attributes and configured attribute-mapping aliases win over promoted metadata. | + +For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes. +Leading or repeated dots, whitespace, other punctuation, and glob expressions +such as `"nv.*"` are rejected. + +Matching is case-sensitive and compares the beginning of each key literally. +Relay does not infer a dot or metadata-key segment boundary. For example, +`"nv."` selects keys that begin with `nv.`, `"nv_"` selects keys that begin +with `nv_`, and `"user"` selects keys such as `username` and `user_api_key`. +Configure the narrowest prefix that selects the metadata you intend to export. + +Scope-start and Scope-end are separate Event records, and Relay runs the +ordered injector chain independently for each record. When Scope-end completes +the span, a metadata key present on that Event replaces the corresponding +promoted Scope-start value. Mark metadata is promoted to the attributes of the +projected span event or tool span. The `gen_ai` projection continues to omit +Marks. + +Promotion supports strings, booleans, signed 64-bit integers, floating-point +numbers, empty arrays, and homogeneous arrays of those primitive types. Relay +omits rejected values and records a bounded runtime diagnostic containing the +key and reason, but it does not record the rejected value or stop trace export. + +Projection-owned attributes and configured attribute-mapping aliases take +precedence over promoted metadata with the same key. Relay also omits selected +keys in namespaces owned by Relay or supported semantic projections: +`nemo_relay.`, `gen_ai.`, `error.`, `exception.`, `input.`, `output.`, `llm.`, +`openinference.`, `server.`, `service.`, `session.`, `tool.`, `tool_call.`, and +`user.`. Relay omits the bare `metadata` key as well. Rejected keys produce a +rate-limited operational diagnostic without dropping the Event or span. + +Promotion does not modify the Event or ATOF payload. Use `resource_attributes` +instead for static values that must be attached to every span from an endpoint. ## Log and Metric Endpoint Resolution