docs: document Event metadata injection

docs/about-nemo-relay/concepts/middleware.mdx

@@ -9,12 +9,13 @@ import { MermaidStyles } from "@/components/MermaidStyles";
 SPDX-License-Identifier: Apache-2.0 */}
 
 This page explains the runtime behavior that runs around managed tool and LLM
-calls and sanitizes emitted mark and scope events.
+calls, injects Event metadata, and sanitizes emitted mark and scope events.
 
 ## What Middleware Is
 
-Middleware controls or transforms tool and LLM execution and sanitizes emitted
-events. NeMo Relay applies each middleware type at a specific lifecycle point.
+Middleware controls or transforms tool and LLM execution and enriches or
+sanitizes emitted events. NeMo Relay applies each middleware type at a specific
+lifecycle point.
 
 Middleware is organized by lifecycle meaning rather than as one undifferentiated
 hook system.
@@ -27,8 +28,9 @@ registrations accept callbacks that return a value or an awaitable when invoked
 through an asynchronous Relay API or queued event publication. Worker and
 native-plugin middleware can also complete asynchronously. Within each
 middleware chain, Relay awaits entries sequentially in priority order so later
-callbacks observe earlier middleware output. Payload and event sanitizer chains
-run on the queued publication path and do not delay managed execution.
+callbacks observe earlier middleware output. Payload sanitizers, Event metadata
+injectors, and Event sanitizer chains run on the queued publication path and do
+not delay managed execution.
 
 The experimental raw C FFI and Go binding retain synchronous middleware
 callbacks. Relay invokes each callback on a native thread and waits for it to
@@ -49,10 +51,11 @@ sanitization and publication rather than awaiting it. Manual lifecycle APIs
 synchronous and create or close their handle immediately.
 
 <Warning>
-Event sanitizers, conditional-execution guardrails, request intercepts,
-execution intercepts, and subscribers are not re-entrant. These callbacks must
-not invoke another NeMo Relay API that runs middleware, flushes subscriber
-delivery, waits on an exporter, or clears plugins. Scope APIs remain supported:
+Event metadata injectors, Event sanitizers, conditional-execution guardrails,
+request intercepts, execution intercepts, and subscribers are not re-entrant.
+These callbacks must not invoke another NeMo Relay API that runs middleware,
+flushes subscriber delivery, waits on an exporter, or clears plugins. Scope
+APIs remain supported:
 callbacks may create, push, or pop scopes at any nesting level and may replace
 the active scope stack with an arbitrary stack. Emitting a new event is the
 only supported operation that can enqueue additional callback work; Relay
@@ -83,8 +86,8 @@ separately from this application-visible result.
 
 ## Registration Levels
 
-Middleware and subscribers can be registered at different levels depending on their
-lifetime and visibility.
+Middleware and subscribers can be registered at different levels depending on
+their lifetime and visibility.
 
 ### Global Registrations
 
@@ -107,10 +110,11 @@ everything in application code.
 
 ## Middleware Families
 
-NeMo Relay has two major middleware families with three distinct purposes:
+NeMo Relay has three major middleware families with four distinct purposes:
 
 - **Intercepts** change the real request or callback execution path.
 - **Conditional-execution guardrails** decide whether the real work runs.
+- **Event metadata injectors** add flat metadata to emitted events.
 - **Sanitize guardrails** change emitted observability without changing the real
   request or result.
 
@@ -122,6 +126,8 @@ Choose the middleware type that matches the behavior you need:
   rejected.
 - Use a **request intercept** when the real request must change before the call.
 - Use an **execution intercept** when code must run before or after the callback.
+- Use an **Event metadata injector** when every emitted Event should receive
+  additional flat metadata before sanitization.
 - Use a **sanitize guardrail** when only subscribers and exporters should see
   rewritten data.
 - Use a **mark or scope event sanitizer** when the sensitive fields are in
@@ -144,6 +150,10 @@ execution intercept cannot undo work from a `next` continuation that it already
 invoked. This fail-closed contract is the same whether the callback completes
 directly or asynchronously.
 
+Event metadata injectors fail open. If an injector fails, panics, or returns an
+invalid attribute set, Relay logs the failure, omits all metadata from that
+injector for the current Event, and continues with the original Event.
+
 ## Intercepts
 
 Intercepts are middleware that change the real request or execution path.
@@ -177,6 +187,57 @@ LLM streaming has a stream execution path for wrappers that need to run around
 chunk delivery and finalization rather than only around a single response
 object.
 
+## Event Metadata Injection
+
+Event metadata injectors add OTel-compatible values to the existing Event
+`metadata` object. They run on every delivered Event and receive the current
+Event as immutable context, so a callback can inspect the Event kind and name
+before deciding what to return.
+
+The v1 contract accepts keys made of ASCII letter, number, underscore, or
+hyphen segments separated by optional single dots. Empty keys, whitespace,
+unsupported punctuation, leading or trailing dots, and repeated dots are
+invalid. Relay stores accepted keys literally; dots do not create nested
+objects. Values may be strings, numbers, booleans, or homogeneous lists of
+those primitive types. Nested objects, `null`, and mixed-type lists are rejected
+as one atomic callback result.
+
+Injection is insert-only. Existing Event metadata is never overwritten.
+Injectors run in ascending numeric priority and then by registration name, so
+the first injector to insert a missing key wins. A later injector that returns
+the same key cannot replace it.
+
+The following Rust registration adds one string to matching Events:
+
+```rust
+use std::collections::BTreeMap;
+use std::sync::Arc;
+
+use nemo_relay::api::registry::register_event_metadata_injector;
+use serde_json::json;
+
+register_event_metadata_injector(
+    "machine-profile",
+    10,
+    Arc::new(|event| {
+        Box::pin(async move {
+            let attributes = if event.name().starts_with("tool") {
+                BTreeMap::from([("nv.machine.profile".into(), json!("dgx"))])
+            } else {
+                BTreeMap::new()
+            };
+            Ok(attributes)
+        })
+    }),
+)?;
+```
+
+Register injectors globally, on an owning scope, or through a plugin
+registration context. Injectors execute on the queued serial publication path,
+before Event sanitizers, so they do not delay the managed tool or LLM callback.
+Sanitizers retain the final decision to preserve, rewrite, or remove injected
+metadata before subscriber and exporter delivery.
+
 ## Guardrails
 
 Guardrails are middleware that block execution or sanitize observability payloads.
@@ -229,16 +290,17 @@ arguments passed to the callback or the real value returned to the caller.
 ## Queued Event Publication
 
 Scope operations, marks, and manual or managed tool/LLM lifecycle calls do not
-await observability sanitizers. At emission time Relay snapshots the event-only
-payload, visible sanitizer chains, and subscribers, then places the work on a
-serial dispatcher. The dispatcher awaits the specialized tool or LLM payload
-sanitizers, then the event sanitizers, and publishes the event later in FIFO
-order.
+await observability middleware. At emission time Relay snapshots the event-only
+payload, visible Event metadata injectors, sanitizer chains, and subscribers,
+then places the work on a serial dispatcher. The dispatcher awaits specialized
+tool or LLM payload sanitizers, applies Event metadata injectors, runs Event
+sanitizers, and publishes the Event later in FIFO order.
 
 Subscriber and exporter delivery is therefore delayed, while start/end/mark
 order is preserved. Closing a scope or deregistering middleware after emission
-does not affect queued snapshots. Sanitizer failures fail closed: Relay records
-the callback failure and withholds the governed observability payload.
+does not affect queued snapshots. Injector failures fail open and omit that
+injector's output. Sanitizer failures fail closed: Relay records the callback
+failure and withholds the governed observability payload.
 
 ## Managed Execution Order
 
@@ -273,10 +335,13 @@ flowchart LR
 
     subgraph PublicationPath[Serial Publication Path]
         RequestSanitizers[Request Sanitizers]
+        StartInjection[Event Metadata Injectors]
         StartEvent[Scope-Start Sanitizers and Deliver Start]
         ResponseSanitizers[Response Sanitizers]
+        EndInjection[Event Metadata Injectors]
         EndEvent[Scope-End Sanitizers and Deliver End]
-        RequestSanitizers --> StartEvent --> ResponseSanitizers --> EndEvent
+        RequestSanitizers --> StartInjection --> StartEvent
+        StartEvent --> ResponseSanitizers --> EndInjection --> EndEvent
     end
 
     QueueStart -.-> RequestSanitizers
@@ -292,8 +357,8 @@ The phases run as follows:
 3. **Response phase:** Relay snapshots and enqueues the end event's
    observability copy, then returns the real result.
 4. **Publication path:** The serial dispatcher runs request or response
-   sanitizers, then the matching scope-event sanitizers, and finally delivers
-   the event to subscribers and exporters.
+   sanitizers, applies Event metadata injectors, runs the matching scope-event
+   sanitizers, and finally delivers the Event to subscribers and exporters.
 
 The start event is submitted before execution begins, but its sanitizers run
 later on the publication path and may overlap application execution. The serial
@@ -303,6 +368,7 @@ for queued sanitization and delivery when a caller needs that barrier.
 This ordering preserves the distinction between the families:
 
 - Use an intercept to change real execution.
+- Use an Event metadata injector to add observability context.
 - Use a sanitize guardrail to change only emitted observability.
 
 ### Rejection Path