Skip to content

metric sinks: SHOW, catalog visibility, and RBAC (SQL-572) - #38149

Draft
mtabebe wants to merge 4 commits into
MaterializeInc:mainfrom
mtabebe:ma/prom-metrics/sql-572-show-rbac
Draft

metric sinks: SHOW, catalog visibility, and RBAC (SQL-572)#38149
mtabebe wants to merge 4 commits into
MaterializeInc:mainfrom
mtabebe:ma/prom-metrics/sql-572-show-rbac

Conversation

@mtabebe

@mtabebe mtabebe commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Problem:

A metric sink was invisible, mz_objects did not carry it, and there
was no way to read back the SQL that created it. Creating one required
ownership of the FROM relation, stricter than any other sink.

Solution:

SHOW METRIC SINKS [FROM <schema>] [IN CLUSTER <c>] [LIKE ...] lists
name, from, and cluster. SHOW [REDACTED] CREATE METRIC SINK replays
create_sql. Both require enable_metric_sink.

mz_internal.mz_metric_sinks exposes what the catalog knows about a
sink, shaped like mz_catalog.mz_sinks, and joins into mz_objects as
type metric-sink. It is a materialized view derived from
mz_catalog_raw.

Create now needs CREATE on the schema, CREATE on the cluster, and
read on the FROM relation, the same shape as CREATE SINK. You need
read on it, not ownership.

The relation carries only the columns something reads. create_sql and
redacted_create_sql have no reader, and SHOW CREATE is how you get
the SQL back.

Testing:

  • metric_sink.slt covers discovery and access control: the
    mz_metric_sinks row resolving to its FROM relation, cluster, schema,
    and owner, mz_objects and mz_show_all_objects membership, the three
    SHOW METRIC SINKS filters, the SHOW CREATE round-trip, the audit
    rows, and privileges.

  • The restart platform check reads SHOW METRIC SINKS instead of
    probing with a CREATE expected to fail.

mtabebe and others added 3 commits August 10, 2026 16:37
@mtabebe @antiguru
sql: add CREATE METRIC SINK as a durable catalog object (SQL-554)
01f5b19 
Problem:

To maintain cluster metrics we defined the MetricSink compute operator.
However, there is no way to ask for one. We need a SQL surface that
names a metric sink and a durable catalog representation that survives
a restart.

Solution:

Add `CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel>` and `DROP
METRIC SINK`, gated behind `enable_metric_sink`. Creating a sink writes a
catalog item and nothing else: no dataflow is optimized or shipped, so a
sink created today publishes no metrics.

Planning checks that the `FROM` relation exposes the five columns the
operator reads (`metric_name`, `metric_type`, `labels`, `value`, `help`).
Note: order is not enforced and extra columns are fine. Nullability is not
checked.

Metric sinks need no new durable record. They persist as ordinary `Item`s
and `item_type` works the type out from `create_sql`.

However, the changes to audit and serialization does bump the catalog
version.

There are some gaps in:
- `mz_comments` has no `MetricSink` branch, but `CommentObjectType`
has no variant either, so `COMMENT ON METRIC SINK` does not parse
and no such record can exist
- `MZ_DEFAULT_PRIVILEGES` has no CASE arm, but `ON METRIC SINKS`
is rejected during planning

Note: `enable_metric_sink` is off by default

Testing:

- New `test/sqllogictest/metric_sink.slt`: the column contract and each
 way of violating it, `FROM` targets with no rows to read, `IF NOT
EXISTS`, the flag-off refusal, and the seven views above answering with
 a metric sink present.

Co-Authored-By: Moritz Hoffmann <mh@materialize.com>
@mtabebe @claude
sql: address review comments on metric sink DDL (SQL-554)
136b2ba 
Make CreateMetricSinkStatement.name non-nullable, collapse the redundant
item_type reject match into the relation_desc None branch, and add a
TODO(SQL-572) on the platform-check survival probe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mtabebe @antiguru
metric sinks: run, rehydrate, and track dataflow progress (SQL-571)
a503c81 
Problem:

`CREATE METRIC SINK` only wrote a catalog item. It never
optimized or shipped a dataflow, so nothing actually ran. On top of that
a metric sink export had neither a trace nor a `sink_write_frontier`, so
every frontier report logged `collection without write frontier`, the
controller never saw progress, and the since of the sink's input stayed
pinned.

Solution:

`CREATE METRIC SINK` now optimizes and ships a dataflow, `DROP`
tears it down, and bootstrap re-renders every metric sink so one survives
a restart. The assembly optimizer starts from the `GlobalId` of the
collection to export, like `CREATE INDEX`, pushes the row-wise shaping
into MIR, and exports a single `MetricSink` sink. Sequencing is staged so
optimization runs off the coordinator thread. Bootstrap reuses the global
expression cache, like materialized views do. A cached plan embeds a
transient id for the shaped view, and reusing that id across a boot is
safe because build ids are dataflow-local on the worker and never
registered in the controller's instance-global collections.

For progress, the operator already folds the combined ok+err input
frontier, so we hand that to the `ComputeState`. Every worker reports it,
not just the one that owns the registry collector. The controller meets
the per-worker frontiers, so a worker stuck at the minimum would
otherwise hold the input back.

Rendering create-item notices and shipping a dataflow under a read hold
move into the shared `render_create_item_notices` and `ship_new_dataflow`
helpers, with `CREATE INDEX` moved onto both.

Testing:

testdrive drives a sink's series into the replica's Prometheus
registry and back out through `mz_cluster_prometheus_metrics`, then
checks that `DROP METRIC SINK` and `DROP VIEW ... CASCADE` retract them.

The restart platform check now probes for the re-rendered dataflow, not
just the re-parsed catalog item. Introspection is per-replica and the
checks default to two replicas, so it targets one.

An optimizer unit test pins the assembled dataflow to exactly one
`MetricSink` export over the shaped view.

Co-Authored-By: Moritz Hoffmann <antiguru@gmail.com>
@mtabebe
mtabebe force-pushed the ma/prom-metrics/sql-572-show-rbac branch from 58c8c11 to 2599599 Compare August 11, 2026 13:59
@mtabebe
metric sinks: SHOW, catalog visibility, and RBAC (SQL-572)
5eeeece 
Problem:

A metric sink was invisible, `mz_objects` did not carry it, and there
was no way to read back the SQL that created it. Creating one required
ownership of the `FROM` relation, stricter than any other sink.

Solution:

`SHOW METRIC SINKS [FROM <schema>] [IN CLUSTER <c>] [LIKE ...]` lists
name, `from`, and cluster. `SHOW [REDACTED] CREATE METRIC SINK` replays
`create_sql`. Both require `enable_metric_sink`.

`mz_internal.mz_metric_sinks` exposes what the catalog knows about a
sink, shaped like `mz_catalog.mz_sinks`, and joins into `mz_objects` as
type `metric-sink`. It is a materialized view derived from
`mz_catalog_raw`.

Create now needs `CREATE` on the schema, `CREATE` on the cluster, and
read on the `FROM` relation, the same shape as `CREATE SINK`. You need
read on it, not ownership.

The relation carries only the columns something reads. `create_sql` and
`redacted_create_sql` have no reader, and `SHOW CREATE` is how you get
the SQL back.

Testing:

- `metric_sink.slt` covers discovery and access control: the
`mz_metric_sinks` row resolving to its `FROM` relation, cluster, schema,
and owner, `mz_objects` and `mz_show_all_objects` membership, the three
`SHOW METRIC SINKS` filters, the `SHOW CREATE` round-trip, the audit
rows, and privileges.

- The restart platform check reads `SHOW METRIC SINKS` instead of
probing with a `CREATE` expected to fail.
@mtabebe
mtabebe force-pushed the ma/prom-metrics/sql-572-show-rbac branch from 2599599 to 5eeeece Compare August 11, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mtabebe