mdcode: generate BigQuery property-graph DDL from the Semantic Model IR by libei · Pull Request #269 · GoogleCloudPlatform/knowledge-catalog

libei · 2026-08-04T17:58:00Z

Adds a BigQuery consumer of the Semantic Model IR (PR #258) that emits a single
CREATE OR REPLACE PROPERTY GRAPH over the entities' existing base tables:
NODE TABLES for entities, EDGE TABLES for relationships, and model-level
metrics lowered to inline MEASURE(...) properties.

Relationships cover both:

direct foreign keys — the edge is backed by the source entity's own table; and
many-to-many associations — the edge is backed by a junction table with its
own KEY and edge properties, via an additive optional association block on
the IR's Relationship. This is the only IR change here (+22 lines in ir.ts);
the direct-FK path and the loader are untouched.

Tests

Fixture-driven: a corpus of <fixture>.yaml inputs each paired with a committed
<fixture>.bigquery.golden.sql, so every translation is reviewable as text.
bigquery.test.ts holds only what a loadable fixture cannot express — the
hand-built M:N edge (the open format has no association-table syntax), IR-contract
metric cases, and degenerate inputs — plus a structural invariant guard that parses
the emitted DDL and asserts the single-exposed-property shape BigQuery enforces for
graph measures.

tsc --noEmit clean; 102 tests pass. The generated DDL (including the M:N golden)
was run against a live BigQuery instance and traversed with a GQL MATCH.

Builds on #258 (now merged).

Add a BigQuery consumer of the Semantic Model IR that emits a single CREATE OR REPLACE PROPERTY GRAPH statement over the entities' existing base tables: NODE TABLES for entities, EDGE TABLES for relationships, and model-level metrics lowered to inline MEASURE(...) properties. Each entity's dataSource is emitted verbatim as the table reference -- the IR contract already delivers it fully qualified (the loader normalizes it), so the generator never re-prefixes it with the graph's own project/dataset (those name where the graph is created, not where a source table lives). Relationships cover both direct foreign keys (edge backed by the source entity's table) and many-to-many associations (edge backed by a junction table with its own KEY and edge properties), via an additive optional `association` block on the IR's Relationship. Tests are fixture-driven: a corpus of <fixture>.yaml inputs each paired with a committed <fixture>.bigquery.golden.sql. bigquery.test.ts holds only what a loadable fixture cannot express (the hand-built M:N edge, IR-contract metric cases, and degenerate inputs), plus a structural invariant guard over the emitted measures.

amirhormati · 2026-08-04T22:58:50Z

@@ -0,0 +1,222 @@
+# Test fixture -- AI-first semantics format (v0.2.0.dev0).


What is the difference between this and the other one under OSSIE?

fixtures/ossie/tpcds_semantic_model.yaml is the Apache Ossie reference example, copied verbatim (see fixtures/ossie/README.md) — it verifies the loader parses the canonical upstream spec example faithfully and anchors OSI-schema conformance. tpcds_date_edge.yaml is hand-authored in the same format, trimmed to isolate generator edge cases: a relationship whose from/to columns share a name, a keyless target dimension (date_dim) with only custom_extensions, and a second non-BigQuery dialect variant to exercise dialect selection. Added a header note spelling this out.

amirhormati · 2026-08-04T23:19:34Z

 }

+/**
+ * An association (junction) table backing a many-to-many relationship.


Why does many-to-many need a special table?

A many-to-many link can't be a foreign key: an FK column holds a single value, so it references at most one row — a to-one direction. To record that each student takes many courses and each course has many students, the pairs have to live in their own junction table (one row per (source, destination)), which can also carry edge attributes like a grade. Expanded the Association doc comment to state this.

Makes sense.

dlychagin-gg · 2026-08-04T22:24:53Z

+// BIT_AND/OR/XOR all rejected). A metric using an aggregate outside this set
+// therefore cannot be a MEASURE and is skipped + warned rather than emitted as
+// DDL BigQuery would reject.
+const SUPPORTED_AGGREGATES = ['SUM', 'AVG', 'COUNT', 'MIN', 'MAX'];


curious, is DISTINCT aggregate modifier supported with these aggregates in BQ measures?

Yes — verified live on BigQuery: MEASURE(COUNT(DISTINCT p)) and MEASURE(SUM(DISTINCT p)) are both accepted (over an exposed node property). The generator already emits the modifier — see the agg.distinct branch in placeMetric, exercised by the DISTINCT fixtures.

dlychagin-gg · 2026-08-04T22:25:53Z

+  const metrics = model.metrics ?? [];
+
+  if (!entities.length) {
+    warnings.push(


why warning? The IR in this state is invalid (i.e. if it has no entities), so we should just fail here.

Changed — the generator now throws instead of warning when no entity can form a node. An empty NODE TABLES () block is invalid DDL under any circumstance, so we fail loudly rather than return a graph BigQuery would reject. The per-entity skip reasons are folded into the error message.

dlychagin-gg · 2026-08-04T22:32:23Z

+  });
+  if (entities.length && !validEntities.length) {
+    warnings.push(
+        'every entity was skipped (empty KEY); the generated graph would be empty and invalid');


Should we fail here, since the graph will be empty and therefore invalid?

Changed — this now throws. When every entity is skipped for an empty KEY the graph would be empty and invalid, so generatePropertyGraph fails with a message that lists which entities were dropped and why.

dlychagin-gg · 2026-08-04T22:39:44Z

+  // that is all the model carries (a transpile pass may fill `expression`
+  // later).
+  const expression = metricExpression(metric);
+  if (expression === undefined) {


should we fail here? The IR is in an invalid state at this point. (each metric must contain at least one expression)

These per-element cases (metric with no expression, metric on an unknown/keyless entity, an edge with an unknown endpoint) drop a single element and leave the rest of the graph valid, so the generator degrades + warns rather than aborting a whole model over one bad metric. Note a metric with no expression is unreachable from a loaded model — the loader's zod schema requires expression — so this only guards hand-built IR. Per your comment on keyless_dimension, the plan is a strict validate gate (already flagged at loader.ts) that promotes these warnings to hard errors in a fail-fast mode; I'd rather add that one switch than scatter throws. The genuinely-invalid-output case (empty NODE TABLES) is the exception and now throws unconditionally.

dlychagin-gg · 2026-08-04T22:45:30Z

+  // expression; fall back to the declared attach entity when the IR provides
+  // one, so it can still be placed rather than dropped as "references no
+  // entity".
+  if (referenced.length === 0 && metric.entity) {


I thought metric.entity was originally set based on entities referenced by the metric expression. so if referenced.length == 0 then metric.entity is always undefined, therefore line 202 will never be executed.

It's live for hand-built IR, not dead. An author can set metric.entity on a COUNT(*) metric to attach it — that's exactly this fallback, and it's exercised by the two COUNT(*)-lowering tests in bigquery.test.ts (removing the branch made them fail). You're right it never fires for a loaded model: the loader derives metric.entity from the expression's qualifiers, so COUNT(*) (no qualifier) leaves metric.entity unset. Added a comment stating exactly this.

dlychagin-gg · 2026-08-04T23:35:42Z

@@ -0,0 +1,20 @@
+CREATE OR REPLACE PROPERTY GRAPH `sqlgen-testing.demo.lineitem`
+NODE TABLES (


this graph is invalid, we should not produce it.

also, there's no .yaml for this testcase, perhaps we should just remove this .sql file

Agreed on (1) — fixed by making the generator throw on an empty graph, so this DDL can no longer be produced; removed the golden and dropped the fixture from the generator CORPUS. On (2): the .yaml does exist (lineitem_databricks_ext.yaml) and is exercised by loader.test.ts (the unique_keys-without-primary_key path) plus two loader-behavior tests in the e2e file, so I kept it as a loader fixture rather than deleting it.

dlychagin-gg · 2026-08-04T23:40:53Z

+    )
+);
+
+-- warnings --


at some point, we should probably have an execution mode where we'll fail instead of reporting warnings. A warning for an unsupported dialect is fine, but maybe at some point we should have a mode where we'd fail if we start ignoring entities or metrics. wdyt?

Agreed — that's the strict validate gate (already noted at loader.ts). Default stays lenient: an unsupported dialect is a warning, and dropping an entity/metric is a warning; in strict mode those become errors. This PR keeps that default but makes the one always-invalid case — an empty graph (no node tables) — a hard failure regardless of mode.

dlychagin-gg · 2026-08-04T23:47:27Z

@@ -0,0 +1,24 @@
+CREATE OR REPLACE PROPERTY GRAPH `sqlgen-testing.bei_semantic_ir_verify.school_graph`


where is yaml for this golden?

There's no .yaml because OSI can't express a junction table — this golden is generated from hand-built association IR in bigquery.test.ts (the M:N/association path), then validated against live BigQuery. Same reason as the junction-table thread above.

dlychagin-gg · 2026-08-04T23:47:43Z

@@ -0,0 +1,31 @@
+CREATE OR REPLACE PROPERTY GRAPH `sqlgen-testing.demo.sales`


where is yaml for this golden?

It's right next to the golden — star_orders_customer.yaml — and it's the first entry in the e2e CORPUS list, so the golden loop regenerates from it.

dlychagin-gg · 2026-08-04T23:49:49Z

@@ -0,0 +1,49 @@
+CREATE OR REPLACE PROPERTY GRAPH `sqlgen-testing.demo.vendor_sales`


where is yaml for this golden?

It exists — vendor_dialects.yaml, next to the golden and in the e2e CORPUS list.

- fail (throw) when no valid node table remains, instead of emitting an invalid empty NODE TABLES block; drop the now-unproducible lineitem_databricks_ext generator golden (kept the loader fixture) - placeMetric takes the skipped-entity set instead of re-deriving the keyless check - rename splitAggregate -> extractAggregate - note that the COUNT(*) metric.entity fallback is the hand-built-IR path - expand the Association and tpcds_date_edge fixture doc comments

libei requested a review from amirhormati August 4, 2026 17:58

libei force-pushed the upstream-pr2-bigquery-graph branch from 7b6ba24 to 293f42e Compare August 4, 2026 20:11

libei force-pushed the upstream-pr2-bigquery-graph branch from 293f42e to 1b8fa0c Compare August 4, 2026 20:56

libei mentioned this pull request Aug 4, 2026

mdcode: add semantic-model scope and BigQuery Graph push #270

Merged

amirhormati reviewed Aug 4, 2026

View reviewed changes

amirhormati approved these changes Aug 4, 2026

View reviewed changes

dlychagin-gg reviewed Aug 4, 2026

View reviewed changes

libei marked this pull request as ready for review August 5, 2026 05:08

libei merged commit 930b65f into GoogleCloudPlatform:main Aug 5, 2026
7 checks passed

		@@ -0,0 +1,222 @@
		# Test fixture -- AI-first semantics format (v0.2.0.dev0).

		@@ -0,0 +1,20 @@
		CREATE OR REPLACE PROPERTY GRAPH `sqlgen-testing.demo.lineitem`
		NODE TABLES (

		@@ -0,0 +1,24 @@
		CREATE OR REPLACE PROPERTY GRAPH `sqlgen-testing.bei_semantic_ir_verify.school_graph`

		@@ -0,0 +1,31 @@
		CREATE OR REPLACE PROPERTY GRAPH `sqlgen-testing.demo.sales`

		@@ -0,0 +1,49 @@
		CREATE OR REPLACE PROPERTY GRAPH `sqlgen-testing.demo.vendor_sales`

Uh oh!

Conversation

libei commented Aug 4, 2026

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants