Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The
- **Langfuse parallel-branches mapping parity** (proposal 0088, observability §8.4.8 / §8.3 / §8.4.2 / §3.4, spec v0.83.0). Brings the Langfuse observer's parallel-branches rendering to parity with the OTel side. The observer already synthesized the three-level Observation tree (the parallel-branches node Span, a per-branch dispatch Span named by the `branch_name`, and the branch's inner observations) and already emitted the dispatch-span `parallel_branches_parent_node_name` and `branch_name`; the two node-span attributes `parallel_branches_branch_count` and `parallel_branches_error_policy` are now flattened onto the node Span's `observation.metadata` (mirroring the `fan_out_*` attributes), the one §8.4.2 row the observer had never mapped. The three `parallel_branches_*` keys join the reserved caller-metadata set (26 to 29), so a caller passing one as invocation metadata is rejected at the `invoke()` boundary rather than shadowing the OA-emitted field. The OTel side was already complete. Conformance fixture 136 (the dedicated three-level-tree pin) is un-deferred; fixture 030's incidental coverage stands.
- **Adaptive call-level retry: per-attempt request override** (proposal 0095, llm-provider §7.1, spec v0.91.0). The LLM-completion call-level retry loop gains an opt-in per-attempt request override. A new `LlmRetryConfig` (the llm-provider-scoped superset of the generic `RetryConfig`, exported from `openarmature.llm`) carries a `per_attempt_override`: a schedule of `RuntimeConfig` partials applied to retries. Attempt 0 uses the caller's base `config` unchanged; retry `i` merges `per_attempt_override[i]` onto the base (the override's non-None fields replace; a None or unspecified field inherits the base, per the §6 null-skip semantics), and the last entry carries forward when the schedule is shorter than the retry count. The canonical use is an escalating temperature schedule that breaks the "temperature 0 replays the same output" determinism trap on a retried structured-output call. `complete()` never mutates the caller's `config` (each attempt config is a fresh copy), and a plain `RetryConfig` preserves the existing byte-identical replay. The per-attempt OTel span carries a new `openarmature.llm.retry_reason` attribute (`transient`) on retries, absent on the base attempt. This is the first half of proposal 0095; the structured-output reask half follows. Spec v0.91.0 is beyond the current v0.88.0 pin, so the behavior ships ahead of the pin (unit-tested); the conformance fixtures 061-066 ride the v0.17.0 pin bump.
- **Adaptive call-level retry: structured-output reask** (proposal 0095, llm-provider §7.1, spec v0.91.0). The second half of 0095. `LlmRetryConfig` gains an opt-in `reask` builder (`Callable[[StructuredOutputInvalid], str]`). When present, a `structured_output_invalid` failure becomes retryable for that call (a call-level convenience, not a classifier change; without a builder it stays non-transient and raises on the first occurrence). On each such failure the loop appends two messages to a working transcript, the model's raw output as an `assistant` message and the builder's returned correction as a `user` message, so the retry is informed rather than a byte-identical replay. OA authors no prompt of its own (the caller owns every word beyond the model's output); the builder receives the raised `StructuredOutputInvalid` (its `raw_content` and `failure_description`). The transcript accumulates reask pairs across reask retries and consumes the `max_attempts` budget; a transient retry interleaved in a reask loop re-sends the accumulated transcript unchanged. `complete()` never mutates the caller's `messages` (each reask replaces the transcript with a fresh list rather than appending in place). The retry span's `openarmature.llm.retry_reason` is `reask` on a reask retry, `transient` otherwise. A reask always appends the model output as a fresh `assistant` message (never continues a trailing one): §3 requires the last message before a call to be `user`/`tool`, so the transcript never ends in `assistant`. Ships ahead of the pin (unit-tested); fixtures 062-066 ride the pin bump.
- **Langfuse observer: credentials-in construction with tracer-provider isolation** (proposals 0114 + 0116 + 0117, observability §6 / §8.9, spec v0.108.0 / v0.110.0 / v0.111.0). The Langfuse observer gains a second construction mode alongside today's caller-supplied client: `LangfuseObserver.from_credentials(public_key=..., secret_key=..., host=...)` (over the lower-level `LangfuseSDKAdapter.from_credentials(...)`) builds an OA-owned `Langfuse` client on a dedicated `TracerProvider` by default, so its observations no longer bind the global provider and leak onto the application's OTel backend. A Langfuse v4 client constructed with no `tracer_provider=` attaches its span processor to the globally-registered provider, so in any service that registers a global provider (the standard app-tracing setup) attaching the Langfuse observer silently exported every observation, prompts and completions included, to the app backend. Because the Langfuse SDK caches one client per `public_key`, a dedicated provider takes effect only when OA is the first constructor for that credential; OA reuses one isolated provider per credential and reads the actual binding back after construction. The invariant covers every payload OA harvests from the runtime -- the provider payload (`disable_provider_payload`), the Trace-level state input/output (`disable_state_payload` and the `trace_input_from_state` / `trace_output_from_state` hooks), and a failed Tool / Embedding / Retriever / LLM observation's `error_message` / `error_type` -- but not the dimensions the caller deliberately attaches (`correlation_id` / `session_id` / `userId` / trace name / caller metadata), which stay verbatim as cross-backend join keys. When any construction-determinable channel is live and OA establishes the client is bound to a provider it did not isolate, construction fails loud with a categorized `LangfuseProviderIsolationUnavailable` before any observation is emitted, rather than leaking payloads to a shared backend; where OA cannot establish the binding at all (a future SDK), it suppresses every channel and logs a warning. The failed-observation error message is gated per-emission (not knowable at construction): on an un-isolatable provider it is omitted, retaining only the error category where one exists (a Tool failure has no category, so it carries no message-derived status either). A single `accept_shared_provider=True` opt-out turns the whole thing into a warn-and-proceed onto the shared provider. With no channel live (the default privacy posture), an un-isolatable client neither raises nor warns. The existing caller-supplied path (mode a) is unchanged and never mutated: a caller who builds their own client stays responsible for isolating its `tracer_provider`, and OA documents the remedy rather than reaching into the supplied client. The `secret_key` is accepted as a `pydantic.SecretStr`, masked in OA's own reprs and logs with the plaintext read only at the SDK call (`public_key` and `host` stay plain strings), and a blank credential is rejected at the boundary rather than falling through to the SDK's ambient `LANGFUSE_*` environment fallback. A `sample_rate` passed for the client is applied to the isolated provider, since the SDK only honors it on a provider it builds itself. `accept_shared_provider` binds the provider the application already registered rather than letting the SDK construct and globally register one of its own, which would capture OTel's single-assignment global slot. The new `LangfuseProviderIsolationUnavailable` derives from an `ObservabilityError` base, a fourth hierarchy alongside the graph-engine, llm-provider, and checkpoint ones. Spec v0.108.0 / v0.110.0 / v0.111.0 are beyond the current v0.107.0 pin, so this ships ahead of the pin (unit-tested); the conformance fixtures (157 / 158, proposals 0115 / 0116 / 0117) ride the pin bump. The LLM error-message arm ships ahead of its spec formalization (proposal 0118, in progress at time of writing).

### Changed

Expand All @@ -35,6 +36,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The

### Fixed

- **The Langfuse failure-isolation marker no longer carries the caught exception's message** (proposal 0118, observability §8.4). **Behavioral for the Langfuse mapping.** The `openarmature.failure_isolated` marker span wrote the caught exception's message into `observation.metadata.error_message`, but that span is a graph-mechanism marker no §8.4.x mapping table covers, so writing harvested exception content onto it was non-conforming over-emission: an exception message that can echo application data (PII, tool arguments, an upstream API error body) reached the Langfuse backend under every privacy setting, since no knob gated it. The marker now carries only `error_category` (plus the caller-supplied `failure_isolation_event_name` and the node name), matching the node span's treatment, so an isolated node failure and an ordinary node failure render the same. The full exception is unaffected on the OTel side, where the failure-isolation span still carries it as the `openarmature.failure_isolation.message` attribute. A sweep of every bundled Langfuse handler against the same rule found no other unmapped harvested-content emission.
- **The OTel `openarmature.llm.complete` span records the exception event on a failed attempt** (observability §4.2). A failed provider-call span carried `ERROR` status but no exception event; the node and invocation spans already recorded it, the LLM span did not. It now records the OTel semconv exception event (`exception.type` / `exception.message`) on every failed attempt, matching the sibling spans. Surfaced while wiring the proposal 0082 error-span fixtures.

## [0.16.0] — 2026-07-18
Expand Down
15 changes: 12 additions & 3 deletions docs/agent/non-obvious-shapes.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,15 +115,24 @@ Different classes, same OTel-Logs export path. If both are attached against the

`install_log_bridge` detects either handler class against the same provider and skips its own `addHandler` accordingly; the `openarmature.correlation_id` LogRecord factory still installs. The check is provider-scoped, so an application that intentionally attaches a handler against a DIFFERENT `LoggerProvider` (a separate logs pipeline) still gets the OA bridge against the OA provider; the helper only dedups when the SAME provider would receive duplicate emissions.

### Three exception hierarchies; know which one your code catches
### `LangfuseObserver.from_credentials` can raise on an un-isolatable provider; construct OA's client first, or opt in

`openarmature` exceptions split across three sibling hierarchies:
The trace-side sibling of the `LoggerProvider` log-bridge gotcha above. When OA constructs the Langfuse client for you (`LangfuseObserver.from_credentials(...)` / `LangfuseSDKAdapter.from_credentials(...)`), it binds the client to a dedicated `TracerProvider` so OA's observations do not leak onto a provider shared with the application's OTel backend. But the Langfuse v4 SDK caches ONE client per `public_key` process-wide: if any client for that key was constructed first (the app called `Langfuse()`, used `langfuse.openai` / `@observe`, or an earlier OA call), the SDK returns the cached client and OA's dedicated provider is silently discarded. So isolation only holds when OA is the FIRST constructor for that credential.

Rather than leak silently, OA detects the binding and fails closed. When a harvested-payload channel is live and OA establishes the client landed on a provider it did not isolate, `from_credentials` raises `LangfuseProviderIsolationUnavailable` at construction, surfacing an init-ordering bug you would otherwise never see. The guarded channels are the provider payload (`disable_provider_payload`) and the Trace-level state input/output (`disable_state_payload` and the `trace_input_from_state` / `trace_output_from_state` hooks). Harvested error content is emitted only on the four mapped provider observations (LLM, Embedding, Tool, Retriever), where a failed observation's `error_message` / `error_type` are omitted per-emission whenever OA could not establish that the client is isolated; LLM, Embedding and Retriever keep their error category as the status message, while a Tool failure has no category at all and so renders `ERROR` with a null status message and no error rows. A graph-mechanism span (a node span, a failure-isolation marker) never carries the exception message on any provider; that detail rides the OTel span instead (`openarmature.failure_isolation.message`), never Langfuse. Caller-attached dimensions (`correlation_id`, `session_id`, `userId`, trace name, your own metadata) are join keys by design and stay verbatim.

Remedies: construct OA's Langfuse client before any other client for that `public_key`; or pass `accept_shared_provider=True` to acknowledge a shared provider (OA warns and proceeds); or build your own client with an isolated `tracer_provider=` and pass it in via the caller-supplied path (`LangfuseObserver(client=LangfuseSDKAdapter(your_client))`), which OA never mutates. Under OA's default privacy posture (no payload channel live), an un-isolatable client is harmless and neither raises nor warns.

### Four exception hierarchies; know which one your code catches

`openarmature` exceptions split across four sibling hierarchies:

- `RuntimeGraphError` (in `openarmature.graph`): node execution failures: `NodeException`, `RoutingError`, `EdgeException`, `ReducerError`, `StateValidationError`. Each has a `category` string matching the spec's canonical error categories.
- `CheckpointError` (in `openarmature.checkpoint`): persistence failures: `CheckpointNotFound`, `CheckpointSaveFailed`, `CheckpointRecordInvalid`, `CheckpointStateMigrationMissing`, `CheckpointStateMigrationFailed`, `CheckpointStateMigrationChainAmbiguous`.
- `LlmProviderError` (in `openarmature.llm`): provider call failures: `ProviderAuthentication`, `ProviderInvalidRequest`, `ProviderInvalidResponse`, `ProviderInvalidModel`, `ProviderModelNotLoaded`, `ProviderRateLimit`, `ProviderUnavailable`, `ProviderUnsupportedContentBlock`, `StructuredOutputInvalid`.
- `ObservabilityError` (in `openarmature.observability.langfuse`): observability backend wiring failures: `LangfuseProviderIsolationUnavailable`. Raised while constructing or driving a backend, not while running a graph or calling a provider.

Catching `Exception` works but is too broad; catching one hierarchy misses the other two. If you want to branch on category strings (e.g., for retry logic), catch the relevant base: `RuntimeGraphError` covers all five spec runtime categories, `LlmProviderError` covers all nine provider categories, `CheckpointError` covers all six checkpoint categories. The `TRANSIENT_CATEGORIES` frozenset in `openarmature.llm` enumerates which provider categories are retriable.
Catching `Exception` works but is too broad; catching one hierarchy misses the other three. If you want to branch on category strings (e.g., for retry logic), catch the relevant base: `RuntimeGraphError` covers all five spec runtime categories, `LlmProviderError` covers all nine provider categories, `CheckpointError` covers all six checkpoint categories. The `TRANSIENT_CATEGORIES` frozenset in `openarmature.llm` enumerates which provider categories are retriable.

### Filter `openarmature.*`-namespaced events when your observer only cares about user nodes

Expand Down
25 changes: 25 additions & 0 deletions docs/concepts/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -1122,8 +1122,32 @@ pip install 'openarmature[langfuse]'

Production wire-up:

```python
from pydantic import SecretStr
from openarmature.observability.langfuse import LangfuseObserver

observer = LangfuseObserver.from_credentials(
public_key="pk-lf-...",
secret_key=SecretStr("sk-lf-..."),
host="https://cloud.langfuse.com",
disable_provider_payload=False,
)
```

Prefer `from_credentials`: openarmature builds the Langfuse client on a
dedicated `TracerProvider`, so its observations do not also land on the
provider your application registered globally. A client you build
yourself binds the global provider unless you pass `tracer_provider=`,
which exports every observation, prompts and completions included, to
your application's tracing backend as well.

If you must build the client yourself (to reuse an existing instance,
say), isolate it explicitly and hand it in; openarmature never mutates
a client you supply:

```python
from langfuse import Langfuse
from opentelemetry.sdk.trace import TracerProvider
from openarmature.observability.langfuse import (
LangfuseObserver,
LangfuseSDKAdapter,
Expand All @@ -1133,6 +1157,7 @@ langfuse_client = Langfuse(
public_key="pk-lf-...",
secret_key="sk-lf-...",
host="https://cloud.langfuse.com",
tracer_provider=TracerProvider(), # keep observations off the global provider
)
observer = LangfuseObserver(
client=LangfuseSDKAdapter(langfuse_client),
Expand Down
20 changes: 10 additions & 10 deletions docs/examples/langfuse-observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,23 +127,23 @@ Wrap the SDK client with `LangfuseSDKAdapter` and pass it to the
observer:

```python
from langfuse import Langfuse
from openarmature.observability.langfuse import (
LangfuseObserver,
LangfuseSDKAdapter,
)
from pydantic import SecretStr
from openarmature.observability.langfuse import LangfuseObserver

langfuse_client = Langfuse(
observer = LangfuseObserver.from_credentials(
public_key="pk-lf-...",
secret_key="sk-lf-...",
secret_key=SecretStr("sk-lf-..."),
host="https://cloud.langfuse.com",
)
observer = LangfuseObserver(
client=LangfuseSDKAdapter(langfuse_client),
disable_provider_payload=False,
)
```

openarmature builds the client on a dedicated `TracerProvider` here, so
its observations stay off the provider your application registered
globally. Building the client yourself binds that global provider
unless you pass `tracer_provider=`, which exports every observation,
prompts and completions included, to your app's tracing backend too.

The adapter bridges `langfuse>=4.6,<5`'s unified `start_observation`
API onto OA's four-method `LangfuseClient` Protocol. v4 has no
explicit trace creation (traces are auto-created from observations);
Expand Down
3 changes: 2 additions & 1 deletion docs/examples/production-observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ part is the observability wiring:
`OTLPSpanExporter` pointed at HyperDX / Honeycomb / Tempo / any
OTLP backend).
- `LangfuseObserver` attached with an `InMemoryLangfuseClient`
(production swaps for `LangfuseSDKAdapter(Langfuse(...))`).
(production swaps for `LangfuseObserver.from_credentials(...)`, which
isolates the client's `TracerProvider`).
- Both observers consume the same `NodeEvent` stream
independently; node code never knows there are two backends.
- `LangfuseObserver` carries `trace_input_from_state` and
Expand Down
Loading