Skip to content

docs: document OpenTelemetry metadata promotion #803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ericevans-nv wants to merge 3 commits into
base: main
Choose a base branch
from
Open

docs: document OpenTelemetry metadata promotion #803

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e7292ab
docs(observability): document metadata promotion
ericevans-nv Aug 18, 2026
2901e8e
docs(observability): clarify literal metadata prefixes
ericevans-nv Aug 18, 2026
136c5bb
docs(observability): align metadata promotion contract
ericevans-nv Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/configure-plugins/observability/opentelemetry.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ type = "gen_ai"
endpoint = "http://localhost:4318/v1/traces"
transport = "http_binary"
service_name = "agent-service"
promote_metadata_prefixes = ["nv."]
max_queue_size = 4096
max_export_batch_size = 512
scheduled_delay_millis = 1000
Expand Down Expand Up @@ -143,6 +144,56 @@ to the other exporters.
| `mark_projection` | `inherit` | Mark representation for `full` and `openinference`: `inherit`, `event`, or `tool`. |
| `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `full` and `openinference` projection. |
| `attribute_mappings` | `[]` | `{ key, alias }` copies applied by `full` and `openinference` projection. |
| `promote_metadata_prefixes` | `[]` | Literal Event metadata prefixes copied to top-level span attributes. |

## Event Metadata Promotion

Set `promote_metadata_prefixes` on a trace endpoint to copy selected keys from
the final sanitized Event metadata into that endpoint's OpenTelemetry output.
The injection and promotion stages use the following contract:

| Behavior | Contract |
|---|---|
| Exact metadata keys | ASCII letters, numbers, underscores, and hyphens in nonempty segments separated by single dots. |
| Promotion prefixes | The same key syntax with an optional trailing dot. Matching is literal and case-sensitive. |
| Values | Strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays containing one supported primitive type. |
| Rejected values | Nulls, objects, nested arrays, mixed-type arrays, and unsigned integers larger than `i64::MAX`. |
| Injector conflicts | Existing Event metadata wins. Otherwise, Relay orders injectors by ascending priority and registration name, and the first successful insertion wins. |
| Scope lifecycle | Relay applies injector precedence independently to Scope-start and Scope-end Events. A Scope-end key is authoritative when Relay constructs the final span; when Scope-end omits the key, the Scope-start value remains. |
| OpenTelemetry collisions | Projection-owned attributes and configured attribute-mapping aliases win over promoted metadata. |

For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes.
Leading or repeated dots, whitespace, other punctuation, and glob expressions
such as `"nv.*"` are rejected.

Matching is case-sensitive and compares the beginning of each key literally.
Relay does not infer a dot or metadata-key segment boundary. For example,
`"nv."` selects keys that begin with `nv.`, `"nv_"` selects keys that begin
with `nv_`, and `"user"` selects keys such as `username` and `user_api_key`.
Configure the narrowest prefix that selects the metadata you intend to export.

Scope-start and Scope-end are separate Event records, and Relay runs the
ordered injector chain independently for each record. When Scope-end completes
the span, a metadata key present on that Event replaces the corresponding
promoted Scope-start value. Mark metadata is promoted to the attributes of the
projected span event or tool span. The `gen_ai` projection continues to omit
Marks.

Promotion supports strings, booleans, signed 64-bit integers, floating-point
numbers, empty arrays, and homogeneous arrays of those primitive types. Relay
omits rejected values and records a bounded runtime diagnostic containing the
key and reason, but it does not record the rejected value or stop trace export.

Projection-owned attributes and configured attribute-mapping aliases take
precedence over promoted metadata with the same key. Relay also omits selected
keys in namespaces owned by Relay or supported semantic projections:
`nemo_relay.`, `gen_ai.`, `error.`, `exception.`, `input.`, `output.`, `llm.`,
`openinference.`, `server.`, `service.`, `session.`, `tool.`, `tool_call.`, and
`user.`. Relay omits the bare `metadata` key as well. Rejected keys produce a
rate-limited operational diagnostic without dropping the Event or span.

Promotion does not modify the Event or ATOF payload. Use `resource_attributes`
instead for static values that must be attached to every span from an endpoint.

## Log and Metric Endpoint Resolution

Expand Down
Loading