Add unified structured logging, tracing, metrics, and OTLP support

[FelineStateMachine]

Summary

Add a Lofi-owned observability surface that gives application authors attractive structured logs by default and carries the same event vocabulary, trace context, and metric definitions across:

• generated browser applications;
• lofi dev, doctor, test, build, and preview;
• framework runtime diagnostics and the development inspector;
• lofi-node daemon, gate, tunnel, Jazz, and iroh boundaries;
• optional OTLP-compatible backends.

The author API should be small and pleasant. OpenTelemetry and runtime-specific exporters remain adapters behind it rather than concepts every application author must configure.

Existing RuntimeDiagnostics and the development inspector remain the immediate, offline source of truthful local state. Telemetry observes and presents those signals; it does not replace them.

Desired author experience

import { telemetry } from "@nzip/lofi";

const log = telemetry.logger("tasks").with({
  component: "TaskList",
});

log.info("Task created", {
  event: "task.created",
  source: "quick-add",
  itemCount: tasks.length + 1,
});

Development output should be compact, colorful, and inspectable:

12:14:08.102  INFO  tasks  Task created
  event=task.created  component=TaskList  source=quick-add  itemCount=4
  trace=7f3c9a…/54b1…

The generated starter should contain one tasteful structured log call. The useful output and easy key/value syntax should encourage good practice without requiring a logging tutorial in the golden path.

Advanced APIs should remain equally small:

await telemetry.span(
  "task.create",
  { "task.source": "quick-add" },
  async () => {
    // application work
  },
);

const created = telemetry.counter("app.task.created", { unit: "{task}" });
created.add(1, { source: "quick-add" });

Architecture

flowchart LR
    A["Author API<br/>logger · span · meter"] --> C["Lofi telemetry core<br/>normalize · context · redact"]
    R["Lofi runtime events"] --> C
    N["lofi-node events"] --> C

    C --> P["Pretty console<br/>browser + terminal"]
    C --> J["JSON lines<br/>CI + daemon logs"]
    C --> M["Memory ring buffer<br/>dev inspector"]
    C --> O["OpenTelemetry API"]

    O --> D["Deno native OTLP<br/>node + tooling"]
    O --> B["Optional browser OTLP/HTTP"]

Loading

Defaults

Runtime	Default presentation	Remote export
Browser development	Pretty console at info; bounded inspector ring buffer	Off
Browser production	warn/error; bounded in-memory diagnostics	Off unless explicitly configured
Lofi commands	Pretty stderr on TTY; JSON in CI/non-TTY	Standard Deno OTLP opt-in
lofi-node start	Pretty stderr on TTY; JSON for daemon/non-TTY	Standard OTEL_* configuration
Tests	Deterministic capture sink; no color or wall-clock IDs	Off

Command output is user interface, not telemetry. Keep dev/doctor tables, command results, and intentional one-time secret presentation separate from operational logging.

Proposed presentation configuration:

• LOFI_LOG_LEVEL=trace|debug|info|warn|error|off
• LOFI_LOG_FORMAT=auto|pretty|json
• respect NO_COLOR
• standard OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_*, propagator, and sampler variables
• retain LOFI_NODE_DEBUG=1 temporarily as a deprecated alias for LOFI_LOG_LEVEL=debug

Do not invent Lofi-specific OTLP configuration where OpenTelemetry already defines a standard variable.

Shared record contract

Create a runtime-neutral normalized record with:

• timestamp and severity;
• instrumentation scope;
• human message plus stable event name;
• flat, typed, bounded attributes;
• normalized error type/message/code/cause/stack;
• active trace/span IDs when available.

Suggested resource identity:

• service.namespace=lofi
• service.name=<app>-web, lofi-cli, or lofi-node
• service.version
• deployment.environment.name
• lofi.runtime=browser|tooling|node
• instrumentation scopes @nzip/lofi, the application name, and @nzip/lofi-node

Use OpenTelemetry semantic conventions where they exist. Reserve lofi.* for framework concepts.

Potential package shape:

• a zero-dependency @nzip/lofi/observability leaf module for records, context, validation, redaction, logger construction, and sink interfaces;
• public convenience export through telemetry from @nzip/lofi;
• no @opentelemetry/sdk-* in the default browser chunk;
• @opentelemetry/api@1 as the trace/metric bridge where a provider is available.

Do not make @std/log the author contract. The facade needs to work consistently in browsers and Deno and must not expose handler/provider configuration to ordinary application code.

Security prerequisite for OTLP

OTLP must not be enabled naively on lofi-node.

Ticket-mode requests contain the app-ticket secret in /t/<secret>/…. Deno's automatic Deno.serve tracing records url.full and url.path. The current gate debug rejection also prints a slice of the ticket-bearing pathname, and lofi-node ticket issue prints the full one-time ticket through console.log, which Deno OTLP console capture can collect.

Before documenting OTEL_DENO=true as supported:

• overwrite the active public-gate span's url.full, url.path, span name, and http.route at the start of the handler, replacing the secret with {ticket};
• replace debug path fragments with a classified route and, only after verification, the non-secret ticket record ID;
• route one-time ticket display through a dedicated sensitive-output path rather than the logger or console.*;
• centrally redact app tickets, node-pairing tickets, recovery phrases, passkey material, admin/backend secrets, authorization headers, query strings, transport URLs containing credentials, and application row payloads;
• run a real local OTLP collector with seeded canaries and fail if a canary appears in any log, span, metric, error, or resource attribute.

This security gate applies to both inbound Deno.serve spans and any browser instrumentation that could observe a credential-bearing Jazz URL.

Initial signal vocabulary

Spans

• lofi.runtime.boot
• lofi.storage.open
• lofi.write.settle
• lofi.sync.reconnect
• lofi.pwa.update
• lofi.node.start
• lofi.node.gate.request
• lofi.node.mesh.connect

Span names must remain low-cardinality; identifiers belong in attributes.

Metrics

• lofi.write.duration
• lofi.write.total
• lofi.write.pending
• lofi.runtime.clients
• lofi.runtime.subscriptions
• lofi.sync.reconnect.total
• lofi.node.gate.requests
• lofi.node.connections
• lofi.node.mesh.rtt

Initial metric attributes should be an explicit low-cardinality allowlist, such as write tier/outcome, sync mode, storage driver, ticket scope, mesh direction, recovery reason, method, classified route, and status class.

App IDs, ticket IDs, trace IDs, row IDs, principals, table contents, dynamic paths, and error messages must not be metric attributes. They may appear only in sanitized logs or spans where operationally necessary.

Implementation plan

1. Contract and Deno/OTLP safety spike

• Add the normalized record model, logger API, attribute validation, error normalization, redaction, context, and sink interfaces.
• Capture the actual Deno 2.9 console-log-to-OTLP shape and confirm trace correlation against a local collector.
• Prove that public gate spans can be sanitized before export.
• Add secret-canary tests covering gate URLs and one-time ticket output.
• Decide from retained evidence whether Deno console capture preserves enough record structure or whether a dedicated optional OTLP log sink is required.

2. Pretty output and author surface

• Implement browser and terminal pretty sinks.
• Implement JSON-lines, deterministic test, and bounded in-memory sinks.
• Snapshot color-stripped and JSON representations from the same normalized record.
• Make handler installation HMR-safe: one handler and one ring buffer after repeated edits.
• Add one structured author log to the generated starter and update the starter snapshot.
• Add a Logs/Spans section to the existing development inspector.

3. Framework runtime instrumentation

• Adapt existing runtime diagnostics into observable gauges rather than maintaining parallel state.
• Add events/spans around boot, durable storage open, local/global write settlement, sync enrollment, foreground recovery, runtime recreation, and PWA updates.
• Instrument transitions and meaningful operations, not render/subscription noise.
• Ensure exporters and sinks never delay or determine local-first behavior.

4. lofi-node adoption

• Replace [gate]/[tunnel] debug helpers and operational console.* with scoped loggers.
• Instrument startup/shutdown, gate decisions, proxy duration, WebSocket handshake/outcome, ticket issue/revoke, pairing, tunnel establishment, Jazz restart, and classified failures.
• Adapt existing iroh connection RTT/path statistics into the shared metric vocabulary.
• Keep command presentation and intentional secret display separate from daemon telemetry.
• Support Deno native OTLP through the standard environment contract once the canary gate passes.

5. Optional browser export

• Keep default browser builds console/ring-buffer only.
• Add OTLP/HTTP through a dynamically loaded explicit opt-in.
• Do not support client-side secret exporter headers.
• Prefer a same-origin relay or a deliberately public constrained ingest endpoint.
• Test CSP, CORS, batching, bounded queues, offline failure, unload/flush behavior, and exporter backpressure.
• Confirm no exporter SDK is included in the default production chunk.

6. Context propagation spike

• Use W3C Trace Context for ordinary HTTP where safe.
• Investigate a verified Jazz/WebSocket or explicit sideband propagation seam.
• Investigate propagation through the custom node-to-node tunnel protocol without exposing context in ticket URLs.
• Until proven, document browser and node traces as structurally consistent but not necessarily one continuous distributed trace.

7. Documentation and golden validation

• Add an author guide focused on the logger/spans/metrics API and examples.
• Add an operator guide for JSON logs and standard Deno OTLP configuration.
• Document privacy, redaction, cardinality, and browser-export constraints.
• Extend generated-project, author-boundary, publication, and golden-path validation.

Acceptance criteria

• Authors can create a scoped logger and emit a message with typed attributes from @nzip/lofi.
• Browser and terminal pretty sinks produce attractive, readable output from the same normalized record used by JSON output.
• Logs emitted in an active span carry correlating trace/span IDs.
• Existing runtime diagnostics feed gauges without creating a second source of truth.
• An exporter outage never delays or fails app boot, local writes, node startup, or normal shutdown.
• Queues and ring buffers are bounded and expose dropped-record counts.
• HMR does not duplicate handlers, records, or inspector buffers.
• Default production browser builds contain no OTLP exporter SDK and make no telemetry network requests.
• Metric instruments enforce an explicit low-cardinality attribute allowlist.
• No seeded secret canary appears in pretty output, JSON, inspector buffers, OTLP logs, spans, metrics, errors, or resource attributes.
• Ticket-bearing paths are sanitized before any automatic HTTP span can be exported.
• One-time ticket output is not emitted through a telemetry-captured console path.
• Existing command-output contracts and generated-project golden tests continue to pass.
• lofi-node start can export sanitized traces, metrics, and correlated logs using standard Deno OTEL_* configuration.
• The docs state truthfully where continuous browser-to-node trace propagation is and is not supported.

Non-goals

• Requiring a collector or observability account for local development.
• Exporting browser telemetry by default.
• Logging schema rows, user-authored content, identity/recovery material, or credential-bearing URLs.
• Deriving metrics automatically from arbitrary log events.
• Building a proprietary telemetry backend.
• Promising end-to-end WebSocket/Jazz trace propagation before a verified transport seam exists.

References

• Deno OpenTelemetry integration
• OpenTelemetry JavaScript status
• OpenTelemetry JavaScript exporters and browser constraints
• OpenTelemetry semantic conventions

[FelineStateMachine]

Sibling wish on the node side: FelineStateMachine/lofi-node#1 (webui dashboard, otlp, slog, schema overview, data freshness, read-only locks). When either gets specced, design them together.