PGO build for releases

[def-]

Adds a driver for building PGO-optimized environmentd and clusterd, plus the
one product change PGO needs before it can collect a usable profile.

bin/pgo-build instrument
<run a representative workload against the instrumented binaries>
bin/pgo-build optimize

Nothing is enabled by default. This is the tooling and the enabler, so the win
can be reproduced and a CI integration argued separately.

Why

Measured on a sqllogictest workload (3371 assertions for training, a disjoint
2453-assertion held-out set), PGO is worth about 6% on the training workload
and about 10% on the held-out one, winning all 8 paired interleaved rounds.
The cost is one extra full build, roughly 17 minutes at a peak of about 19 GB,
which fits an ordinary CI runner.

For comparison, lto = "fat" measured on the same workload gave only 1-3% and
needed an 82 GB link that the OOM killer took on a 125 GB machine. PGO is much
the better trade.

The part that is easy to get wrong

Only a process that exits through its exit handlers writes a usable profile.
Cluster replicas never did, so a training run profiled the parent process and
nothing else, which is a small share of the work: roughly 85% of CPU in this
workload is in the replicas. And a binary built with -Cprofile-use but no
profile data is worse than one left alone, because LLVM treats every function
in it as cold.

Two things fix that, and both are required:

• clusterd handles SIGTERM by exiting normally, compiled in only under
  --cfg pgo_instrument, which bin/pgo-build sets for the instrumented build
  alone. Production shutdown behavior is unchanged.
• Training runs must pass --orchestrator-process-wrapper=env, because
  send_sigterm in mz_orchestrator_process is !command_wrapper.is_empty(),
  so with no wrapper the orchestrator goes straight to SIGKILL, which cannot be
  caught.

optimize warns when it finds fewer profile pools than binaries, since that
failure is otherwise silent.

Testing

No automated tests: this is a build driver, and the product change is #[cfg]-
gated out of every normal build. Verified cargo check -p mz-clusterd both
with and without --cfg pgo_instrument (only the latter type-checks the new
code, and it caught a function_casts_as_integer warning that would otherwise
have reached CI), plus bin/lint-cargo and bin/fmt. Ran the full pipeline
end to end on Linux and confirmed one profile merge pool per instrumented
binary.

BOLT

doc/developer/pgo.md records the BOLT evaluation rather than shipping it.
BOLT does work on top of this, but only after three non-obvious fixes:
restricting it to Rust symbols with --funcs=^_R.* (it cannot reconstruct jump
tables in OpenSSL, librdkafka, protobuf or abseil), dropping
--update-debug-sections (45+ minutes per binary against 2m31s, thanks to
debug = 2), and staging instrumented binaries as real files rather than
symlinks (the orchestrator resolves replicas relative to current_exe(), which
resolves symlinks, so it silently runs the uninstrumented clusterd). With all
that, BOLT adds about 1.3% for roughly 12 more minutes of build and the loss of
core-dump analysis. Written down so nobody re-derives it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Cx3DneVrmS4JYNGmYzbUs7