feat(logs): add pod logs and serverless logs with streaming

[justinwlin]

Closes #323. CON-693.

The problem

There was no way to read logs from the CLI. If a pod failed to start or a worker crash-looped, you had to open the web console, download the logs, and paste them back — which is exactly what agents driving runpodctl were stuck doing. The issue reporter hit this, and so did two UXR sessions.

What this adds

runpodctl pod logs <id>                        # replay recent lines, exit
runpodctl pod logs <id> --follow                # stream until you stop it
runpodctl pod logs <id> --since 30m             # a time window, not a line count
runpodctl pod logs <id> --source system         # platform lines only
runpodctl serverless logs <id>                  # every worker, tagged with workerId
runpodctl serverless logs <id> --worker <wid>   # one worker

Output is JSON Lines — one {source,line,ts} object per line — so it pipes into jq or an agent with no parsing. source is container (your workload's output) or system (the platform narrating image pull, container create/start/stop). A deploy that never comes up usually explains itself in the system lines.

How it works

The v2 REST API serves these as Server-Sent Events. Two things made this more than a wrapper:

These streams never end. They replay tail lines, then stay open forever. There is no EOF, so none of the existing clients in internal/api could be pointed at them — they all do io.ReadAll, which would block until the pod died. This adds the CLI's first streaming transport, and its first REST v2 client (api.runpod.io/v2, RUNPOD_REST_V2_URL). The CRUD commands stay on REST v1; the two hosts are overridden separately so moving one doesn't move the other.

Logs belong to a worker, not an endpoint. serverless logs without --worker resolves the endpoint's workers from the v2 listing and reads them all concurrently, tagging each line. The v1 includeWorkers expansion is not usable for this — it reports EXITED for healthy workers.

Nuances worth knowing

All of these were verified against prod, not inferred from the spec.

• The API sends nothing — not even response headers — until it has a line to send. So a filter matching nothing (--since past the last log, a --source with no output) never answers at all. Measured: ?since=<recent>&source=system sat at 0 bytes for 12s, while since alone answered in 1.5s. That case is reported as a timeout naming both possible causes, deliberately, because exiting 0 with no output would be indistinguishable from a workload that genuinely logged nothing.
• Logs outlive the workload. A stopped pod still returns its full history, including the stop container lines. Post-mortem debugging works, so nothing here is gated on the pod being up.
• --tail is applied per source. --tail 5 on a pod writing both kinds returns up to 5 container lines and up to 5 system lines, so it is not a total.
• --follow reconnects itself and resumes from Last-Event-ID, which is exclusive server-side — so a dropped connection neither duplicates nor skips lines, and it does not replay --tail again. Reconnect notes go to stderr; stdout stays pure JSON Lines.
• A non-follow read is bounded by the client, since the stream has no EOF: it returns once replayed lines stop arriving, or at --max-wait (default 5s).
• The worker set is resolved once at start, so a worker that appears mid-follow needs the command re-run. A crash-looping worker keeps its id across restarts, so the motivating case is covered.
• Concurrent streams are capped at 32, with a note on stderr rather than a silent truncation.

Testing

Full suite green, including -race. 142 assertions across the new packages.

Live verification against prod

Ran against a real pod and a real endpoint, both deleted afterwards:

• snapshot, --tail, --tail 0, --source container, --source system, --since 30m, --since <rfc3339>, -o yaml
• --follow streaming live output, and terminating cleanly on a closed pipe
• serverless logs resolving workers automatically, and --worker explicitly
• a stopped pod (still returns history)
• error paths: bad pod id, bad endpoint id, bad worker id, out-of-range --tail, unknown --source, malformed --since
• confirmed stdout carries only JSON records and errors/notes go to stderr with the existing error codes

Bugs found in review and fixed in the second commit

A review pass over the first commit found several ways a log read could mislead:

• a snapshot discarded any error arriving after --max-wait, so a degraded log store printed nothing and exited 0 — the same observation as a pod with no logs, and a different exit code for the same fault depending only on latency
• a sink error (closed stdout) took the retry branch and hot-looped against the API, and never backed off because each delivered frame reset the backoff
• the resume cursor advanced before a line was delivered, so a failed write let a reconnect resume past a line that was never printed
• one newline-less line grew the process without bound (measured ~400MB); reads are now capped per line and per frame, and the entry is marked truncated
• source/line/ts were omitempty, so a container printing a blank line emitted a record with no line key and jq -r .line returned null
• an endpoint with no workers reported usage_error, which tells an agent its input was wrong; it now reports conflict

Each has a regression test.

[lukepiette]

Read the transport, the fan-in, both commands, and all three test files end to end. This is unusually well-reasoned for a streaming client, and the hard-won details hold up:

• The cursor advancing only after sink delivery (with the regression test pinning it) is the correct resume semantics, and the connected flag genuinely distinguishes "never answered" from "healthy and quiet" — the send-before-streamErr / read-after-receive synchronization is sound, no lock needed.
• readLimitedLine + the frame cap close both shapes of the unbounded-memory problem, and marking entries truncated instead of silently shortening is the right call.
• The sinkError wrapper fixing the hot-loop-on-closed-stdout bug, and the discovery goroutine holding a WaitGroup slot so wg.Add from it can't race close(entries), are both correct and both documented where the next reader needs them.
• Error taxonomy is thoughtful throughout: conflict for a worker-less endpoint (so agents don't re-guess the id), permanent-vs-transient classification shared with the discovery loop, partial fan-in failure not failing the command.
• The AGENTS.md additions encode exactly the on-the-wire behaviors (headers withheld until first line, per-source tail, exclusive Last-Event-ID) that would otherwise be re-discovered the hard way.

Two non-blocking inline notes: the streaming map never shrinks (long-follow cap accounting on churny endpoints), and 2xx responses are assumed to be SSE without a Content-Type check. Neither blocks merge.

Approving.

[lukepiette]

streaming only ever grows — entries are never removed when a stream's goroutine ends. Two consequences on a long --follow of a churning endpoint (workers being replaced get fresh ids, so the map accumulates every id ever seen):

1. atCap counts dead streams against MaxStreams, so after 32 distinct worker ids have existed over the life of the follow, discovery stops attaching new workers even if only a handful of streams are actually live.
2. The "%d streams are already open" note can claim 32 open streams when most have long since 404'd and returned.

The motivating crash-loop case is unaffected (that worker keeps its id), so this is non-blocking — but pruning the map entry when the spawned goroutine returns (or tracking live-count separately from the seen-set used for dedup) would make the cap mean what the stderr note says it means.

[lukepiette]

Any 2xx is handed to the SSE decoder without checking Content-Type. An intermediary that answers 200 with an HTML body (corporate proxy, captive portal, an LB error page) gets decoded as SSE: every line of the page lands on stdout as a raw entry, and since HTML carries no id: fields the cursor never advances, so under --follow the same page replays on every reconnect at the backoff ceiling.

The Raw fallback is the right instinct for a wire-shape change on the real API, but a cheap guard here — if the 2xx Content-Type isn't text/event-stream, return an error naming what came back instead — would separate "the API changed its frame shape" (keep Raw) from "this isn't the API at all" (fail loudly). Non-blocking.