control-plane-api: don't lock injected ops collections during publications

[skord]

Problem

Every publication takes FOR UPDATE on the live_specs rows of ops.us-central1.v1/logs and ops.us-central1.v1/stats (estuary/sre#54). resolve_live_specs injects those two collections into every build so the runtime knows where to write telemetry, and they then land in the passthrough set that verify_unchanged_revisions locks and revision-checks. The result is that all publications across the fleet serialize on the same two rows.

Change

verify_unchanged_revisions now removes the injected ops collections from its expected set before locking. The publication does not modify them and has no correctness dependency on their last_pub_id, so there is nothing to lock or verify:

• The ops collections are still injected into every build; only the locking is skipped.
• A publication which actually drafts an ops collection is unaffected: drafted specs are not passthrough, and are still locked and validated through update_live_specs.
• Removing them from the expected set also removes them from the deleted-since-build check, so no spurious LockFailure is produced for them.

Testing

• New regression test test_injected_ops_collections_are_not_locked: publishes the ops collections, builds an unrelated publication, republishes the ops collections to bump their last_pub_id, then commits the first build. Without the fix the commit fails with a BuildIdLockFailure on the two ops rows (verified); with the fix it succeeds. The test also asserts the ops collections remain present in the build output.
• cargo nextest run -p control-plane-api -p agent: 221 passed, 1 skipped (live Stripe test, skipped in CI as well).

After deploy, the pg_locks blocking tree should no longer show ops.us-central1.v1/logs/stats as the contended live_specs tuples, and concurrent publications of different tenants should no longer wait on each other.

[skord]

Pushed review follow-ups in 53178e6:

• Referenced ops collections are still revision-checked. The skip was previously by name alone, which also dropped the optimistic-concurrency check when a drafted spec genuinely reads an ops collection (as the L2 reporting derivations do). The skip now only applies when the ops collection is not a dependency of any drafted spec in the build.
• Documented on get_ops_collection_names that build-time injection and commit-time lock-skipping must resolve the same set if the lookup ever becomes data-plane-dependent.
• Removed a pre-existing dead line in verify_unchanged_revisions (if !expect_pub_id.is_zero() {}, leftover from an earlier refactor; passthrough rows always carry a non-zero pub id, so the push below it is correctly unconditional).
• Test cleanups, plus new coverage: a derivation reading ops.us-central1.v1/logs now gets a lock failure for exactly that collection when it changes mid-publication, while the merely-injected stats collection remains unlocked.

Full cargo nextest run -p control-plane-api -p agent: 221 passed, 1 skipped (live Stripe test, also skipped in CI).

[jshearer]

This is the critical part:

Never lock or revision-check ops collections which are present only through injection. resolve_live_specs injects ops.us-central1.v1/{logs,stats} into every build so the runtime knows where to write telemetry; the publication does not modify them and has no correctness dependency on their last_pub_id

So long as we can convince ourselves that builds don't actually depend on the contents of the ops collections, but instead just injects them as an implementation detail to enable finding them later in order to write logs/stats, then I think this PR is safe to deploy, otherwise it's not.

The check exists because built specs embed content from the live specs they were validated against, so a passthrough spec changing between build and commit means you'd commit output validated against stale inputs. For that to matter, something must actually consume the embedded copy. For the injected ops collections, nothing does:

• The per-task ops journal templates used at activation are fetched fresh from the current live_specs row at activation time, not from the publication's build DB.
• The runtime's ops publisher doesn't read the ops collection spec either: it writes to labels.LogsJournal/labels.StatsJournal from the shard's labeling (stamped at activation) using a placeholder collection spec.
• Nothing in the commit records ops state: they're not in any task's reads_from/writes_to, so they don't participate in dependency hashes, live_spec_flows, or notify_dependents.

This says to me that we can merge this PR, and also that the real fix is to remove this injection entirely. I asked Claude to do an investigation into the history of its presence and whether we can get rid of it, and the answer appears to be yes:

---

Why the injection exists (June 2024, cpv2). In the cpv2-era runtime, the ops publisher held real *pf.CollectionSpecs: opsLogsSpec/opsStatsSpec fields, ops.ValidateLogsCollection(opsLogsSpec) at startup, with the specs loaded from the task's build DB (cpv2 go/runtime/ops_publisher.go). The reactor genuinely could not route telemetry without the ops collections present in every build, so resolve_live_specs injected them, and the comment "the resulting build will not function properly in the data plane without them" was true when written (commit 3ef202e).

When it stopped being true (August 2024, federated data planes, f1502ba). That work moved ops journal resolution out of the build entirely: activation stamps LOGS_JOURNAL/STATS_JOURNAL shard labels from journal templates fetched fresh from live_specs at activation time, and the runtime was rewritten to write to the label-named journals with a placeholder spec. LogsJournal doesn't exist anywhere in the cpv2 tree and the spec-loading ops publisher doesn't exist in the current one; they're two halves of the same migration. The injection and its comment just survived it.

The decisive evidence that it's vestigial: federated data planes already run without it. A task in ops/dp/public/aws-us-east-1-c1 uses that plane's ops collections via data_planes.ops_logs_name, and those collections are never injected into any build. Meanwhile the pair we do inject (ops.us-central1.v1/*) is the wrong pair for every task outside the legacy plane. So for most of the fleet we've been injecting, fetching, building, and globally locking two collections their builds don't even nominally relate to.

[jshearer]

@jgraettinger I/claude did some investigation and wrote up my findings above ☝️ but this is a "real" change in behavior of the build process, which I am not intimately familiar with. Would you mind taking a look as well?