metric sinks: coordinator per-replica install scaffold (SQL-555) - #38146
Draft
mtabebe wants to merge 5 commits into
Draft
metric sinks: coordinator per-replica install scaffold (SQL-555)#38146mtabebe wants to merge 5 commits into
mtabebe wants to merge 5 commits into
Conversation
Problem: To maintain cluster metrics we defined the MetricSink compute operator. However, there is no way to ask for one. We need a SQL surface that names a metric sink and a durable catalog representation that survives a restart. Solution: Add `CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel>` and `DROP METRIC SINK`, gated behind `enable_metric_sink`. Creating a sink writes a catalog item and nothing else: no dataflow is optimized or shipped, so a sink created today publishes no metrics. Planning checks that the `FROM` relation exposes the five columns the operator reads (`metric_name`, `metric_type`, `labels`, `value`, `help`). Note: order is not enforced and extra columns are fine. Nullability is not checked. Metric sinks need no new durable record. They persist as ordinary `Item`s and `item_type` works the type out from `create_sql`. However, the changes to audit and serialization does bump the catalog version. There are some gaps in: - `mz_comments` has no `MetricSink` branch, but `CommentObjectType` has no variant either, so `COMMENT ON METRIC SINK` does not parse and no such record can exist - `MZ_DEFAULT_PRIVILEGES` has no CASE arm, but `ON METRIC SINKS` is rejected during planning Note: `enable_metric_sink` is off by default Testing: - New `test/sqllogictest/metric_sink.slt`: the column contract and each way of violating it, `FROM` targets with no rows to read, `IF NOT EXISTS`, the flag-off refusal, and the seven views above answering with a metric sink present. Co-Authored-By: Moritz Hoffmann <mh@materialize.com>
Make CreateMetricSinkStatement.name non-nullable, collapse the redundant item_type reject match into the relation_desc None branch, and add a TODO(SQL-572) on the platform-check survival probe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: `CREATE METRIC SINK` only wrote a catalog item. It never optimized or shipped a dataflow, so nothing actually ran. On top of that a metric sink export had neither a trace nor a `sink_write_frontier`, so every frontier report logged `collection without write frontier`, the controller never saw progress, and the since of the sink's input stayed pinned. Solution: `CREATE METRIC SINK` now optimizes and ships a dataflow, `DROP` tears it down, and bootstrap re-renders every metric sink so one survives a restart. The assembly optimizer starts from the `GlobalId` of the collection to export, like `CREATE INDEX`, pushes the row-wise shaping into MIR, and exports a single `MetricSink` sink. Sequencing is staged so optimization runs off the coordinator thread. Bootstrap reuses the global expression cache, like materialized views do. A cached plan embeds a transient id for the shaped view, and reusing that id across a boot is safe because build ids are dataflow-local on the worker and never registered in the controller's instance-global collections. For progress, the operator already folds the combined ok+err input frontier, so we hand that to the `ComputeState`. Every worker reports it, not just the one that owns the registry collector. The controller meets the per-worker frontiers, so a worker stuck at the minimum would otherwise hold the input back. Rendering create-item notices and shipping a dataflow under a read hold move into the shared `render_create_item_notices` and `ship_new_dataflow` helpers, with `CREATE INDEX` moved onto both. Testing: testdrive drives a sink's series into the replica's Prometheus registry and back out through `mz_cluster_prometheus_metrics`, then checks that `DROP METRIC SINK` and `DROP VIEW ... CASCADE` retract them. The restart platform check now probes for the re-rendered dataflow, not just the re-parsed catalog item. Introspection is per-replica and the checks default to two replicas, so it targets one. An optimizer unit test pins the assembled dataflow to exactly one `MetricSink` export over the shaped view. Co-Authored-By: Moritz Hoffmann <antiguru@gmail.com>
mtabebe
force-pushed
the
ma/prom-metrics/sql-555-install
branch
from
August 11, 2026 14:03
6cafcde to
964fb48
Compare
Problem: A metric sink was invisible, `mz_objects` did not carry it, and there was no way to read back the SQL that created it. Creating one required ownership of the `FROM` relation, stricter than any other sink. Solution: `SHOW METRIC SINKS [FROM <schema>] [IN CLUSTER <c>] [LIKE ...]` lists name, `from`, and cluster. `SHOW [REDACTED] CREATE METRIC SINK` replays `create_sql`. Both require `enable_metric_sink`. `mz_internal.mz_metric_sinks` exposes what the catalog knows about a sink, shaped like `mz_catalog.mz_sinks`, and joins into `mz_objects` as type `metric-sink`. It is a materialized view derived from `mz_catalog_raw`. Create now needs `CREATE` on the schema, `CREATE` on the cluster, and read on the `FROM` relation, the same shape as `CREATE SINK`. You need read on it, not ownership. The relation carries only the columns something reads. `create_sql` and `redacted_create_sql` have no reader, and `SHOW CREATE` is how you get the SQL back. Testing: - `metric_sink.slt` covers discovery and access control: the `mz_metric_sinks` row resolving to its `FROM` relation, cluster, schema, and owner, `mz_objects` and `mz_show_all_objects` membership, the three `SHOW METRIC SINKS` filters, the `SHOW CREATE` round-trip, the audit rows, and privileges. - The restart platform check reads `SHOW METRIC SINKS` instead of probing with a `CREATE` expected to fail.
Problem:
The curated metric-sink library runs on every replica and re-renders on
boot, so it cannot be modeled as a durable catalog item the way a user's
`CREATE METRIC SINK`. That would mean per-boot catalog churn and a
builtin migration every time a definition changes.
Solution:
A new `coord/metric_sink.rs` installs each `CURATED` definition on every
replica, mirroring the introspection-subscribe install: bootstrap walks
the existing replicas, replica create installs, replica drop tears down.
Each sink gets a transient `GlobalId` and is recorded in a
`(replica, definition) -> InstalledMetricSink` registry once its dataflow
ships, so a definition that fails to plan or optimize leaves no entry behind.
A replica dropped mid-install is caught by the staged validity recheck
before the finish stage ships.
Sequencing is staged, so optimization runs off the coordinator thread.
The finish stage picks the as-of under a read hold and ships the dataflow
targeted at the replica.
`CURATED` is empty in this PR, so with the flag on or off the system
behaves exactly as today.
To build a dataflow from curated SQL, the metric-sink optimizer's input
becomes a `MetricSinkFrom::{Id, Query}`, mirroring `SubscribeFrom`.
The flag-off path does not clear existing installs on the next replica
restart: command-history replay re-renders a replica-targeted dataflow
on reconnect, so they persist until the replica is dropped or envd restarts.
The operator's health gauges are labelled with the sink's stable
`CuratedMetricSink::name`, not its transient `GlobalId`, so the series stay
identifiable across boots.
Testing:
- An optimizer unit test pins the `Query` source path to the same assembled
shape as the `Id` path: one `MetricSink` export over the shaped view, the
source imported rather than rebuilt.
- A unit test drives `plan_source` against a debug catalog, accepting the
canonical-column contract and rejecting a missing column and a finishing.
- Unit tests cover the `sink` gauge label: a user sink keeps its `GlobalId`,
a curated sink uses its name.
- With `CURATED` empty there is no series to observe, so the install and
teardown hooks are exercised by replica churn in `metric-sink.td` rather
than by an assertion on output.
Co-Authored-By: Moritz Hoffmann <antiguru@gmail.com>
mtabebe
force-pushed
the
ma/prom-metrics/sql-555-install
branch
from
August 11, 2026 14:38
964fb48 to
fb90974
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
The curated metric-sink library runs on every replica and re-renders on
boot, so it cannot be modeled as a durable catalog item the way a user's
CREATE METRIC SINK. That would mean per-boot catalog churn and abuiltin migration every time a definition changes.
Solution:
A new
coord/metric_sink.rsinstalls eachCURATEDdefinition on everyreplica, mirroring the introspection-subscribe install: bootstrap walks
the existing replicas, replica create installs, replica drop tears down.
Each sink gets a transient
GlobalIdand is recorded in a(replica, definition) -> InstalledMetricSinkregistry once its dataflowships, so a definition that fails to plan or optimize leaves no entry behind.
A replica dropped mid-install is caught by the staged validity recheck
before the finish stage ships.
Sequencing is staged, so optimization runs off the coordinator thread.
The finish stage picks the as-of under a read hold and ships the dataflow
targeted at the replica.
CURATEDis empty in this PR, so with the flag on or off the systembehaves exactly as today.
To build a dataflow from curated SQL, the metric-sink optimizer's input
becomes a
MetricSinkFrom::{Id, Query}, mirroringSubscribeFrom.The flag-off path does not clear existing installs on the next replica
restart: command-history replay re-renders a replica-targeted dataflow
on reconnect, so they persist until the replica is dropped or envd restarts.
The operator's health gauges are labelled with the sink's stable
CuratedMetricSink::name, not its transientGlobalId, so the series stayidentifiable across boots.
Testing:
An optimizer unit test pins the
Querysource path to the same assembledshape as the
Idpath: oneMetricSinkexport over the shaped view, thesource imported rather than rebuilt.
A unit test drives
plan_sourceagainst a debug catalog, accepting thecanonical-column contract and rejecting a missing column and a finishing.
Unit tests cover the
sinkgauge label: a user sink keeps itsGlobalId,a curated sink uses its name.
With
CURATEDempty there is no series to observe, so the install andteardown hooks are exercised by replica churn in
metric-sink.tdratherthan by an assertion on output.
Co-Authored-By: Moritz Hoffmann antiguru@gmail.com