Skip to content

mdcode: generate BigQuery property-graph DDL from the Semantic Model IR - #24

Merged
libei merged 2 commits into
upstream-pr1-semantic-ir-loaderfrom
upstream-pr2-bigquery-graph
Aug 2, 2026
Merged

mdcode: generate BigQuery property-graph DDL from the Semantic Model IR#24
libei merged 2 commits into
upstream-pr1-semantic-ir-loaderfrom
upstream-pr2-bigquery-graph

Conversation

@libei

@libei libei commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Second in the capability-layered series merging the semantic-model tooling from the libei/knowledge-catalog fork back into upstream. Stacked on GoogleCloudPlatform#258 (Semantic Model IR + loader); it has no downstream dependencies of its own.

What this PR adds

The first destination for the Semantic Model IR: a generator that emits a single CREATE OR REPLACE PROPERTY GRAPH statement over the entities' existing base tables, with model-level metrics rendered as inline MEASURE(...) properties.

  • src/libts/semantic/bigquery.ts — maps the IR to BigQuery graph DDL:

    • NODE TABLES from entities (KEY from the primary key, fields as properties) and EDGE TABLES from relationships (the association table for an M:N shape, the source base table for a direct FK), each with an explicit element KEY.
    • Measures: each model-level metric is placed on the single entity its aggregate references — a genuinely cross-table aggregate cannot be one MEASURE, so it is skipped and reported. A BigQuery graph measure may only aggregate an exposed property, so each metric is lowered into a derived operand property plus a MEASURE over it; identical operands are exposed once and reused, and COUNT(*) lowers to a count over the key property.
    • Only SUM/AVG/COUNT/MIN/MAX are emitted (the complete set BigQuery accepts inside a measure); any other aggregate is skipped + warned rather than emitted as DDL BigQuery would reject.
    • AI-first metadata (label, temporal-dimension role, description, and the aiContext instructions/synonyms/examples) is folded into OPTIONS(description=...) — the only metadata sink the graph DDL exposes — on the graph, element tables, properties, and measures.
    • Consumes dataSource as the loader's normalized, fully-qualified string; completes an under-qualified ref from options and warns if it is still short.
    • Structural drops (a keyless node, a dangling edge, an unplaceable metric) are surfaced in warnings, never silently emitted as invalid DDL.
  • tests/libts/semantic/bigquery.test.ts — behavior spec over hand-built IR with inline goldens. The chain and M:N goldens are the exact DDL a live BigQuery instance accepted (validated with GRAPH_EXPAND + AGG and a GQL MATCH), plus a parsed-DDL invariant that no emitted MEASURE ever wraps a shape BigQuery rejects.

  • tests/libts/semantic/bigquery.e2e.test.ts + *.bigquery.golden.sql — a file → IR → DDL golden suite over a corpus of real-shaped fixtures; the goldens are the reviewable "big picture" (open a .yaml next to its .bigquery.golden.sql). Regenerate with UPDATE_GOLDENS=1.

Scope

Property-graph DDL generation only. The CLI push path and Knowledge Catalog emit/pull follow in later PRs in the series.

Testing

  • npx tsc --noEmit — clean
  • bun test tests/libts/semantic/ — 109 pass / 0 fail

Review note: this draft is stacked on PR1's branch (upstream-pr1-semantic-ir-loader) within the fork, so the diff shows exactly PR2's additions. When PR1 lands upstream, PR2 will be reopened/retargeted against GoogleCloudPlatform:main.

Adds the first destination for the Semantic Model IR: a generator that emits
a single `CREATE OR REPLACE PROPERTY GRAPH` statement over the entities'
existing base tables, with model-level metrics rendered as inline MEASURE(...)
properties.

- src/libts/semantic/bigquery.ts — maps the IR to graph DDL:
  - NODE TABLES from entities (KEY from the primary key; fields as properties),
    EDGE TABLES from relationships (association table for M:N, source base
    table for a direct FK), with explicit element KEYs.
  - Measures: each model-level metric is placed on the single entity its
    aggregate references (a cross-table aggregate cannot be one MEASURE, so it
    is skipped + warned). A BigQuery measure may only aggregate an EXPOSED
    PROPERTY, so each metric is lowered into a derived operand property plus a
    MEASURE over it; identical operands are exposed once and reused.
  - Only SUM/AVG/COUNT/MIN/MAX are emitted (the complete set BigQuery accepts in
    a measure); anything else is skipped + warned rather than emitted as DDL
    BigQuery rejects. COUNT(*) lowers to COUNT over the key property.
  - AI-first metadata (label, temporal-dimension role, description, and the
    aiContext instructions/synonyms/examples) is folded into OPTIONS(description)
    — the only metadata sink the graph DDL exposes — on the graph, element
    tables, properties, and measures.
  - dataSource is consumed as the loader's normalized, fully-qualified string;
    an under-qualified ref is completed from options and warned if still short.
  - Structural drops (keyless node, dangling edge, unplaceable metric) are
    reported in `warnings`, never silently emitted as invalid DDL.

- tests/libts/semantic/bigquery.test.ts — behavior spec over hand-built IR with
  inline goldens; the two chain/M:N goldens are the exact DDL a live BigQuery
  instance accepted (GRAPH_EXPAND + AGG and a GQL MATCH), plus an invariant that
  no emitted MEASURE ever wraps a shape BigQuery rejects.

- tests/libts/semantic/bigquery.e2e.test.ts + *.bigquery.golden.sql — file -> IR
  -> DDL golden suite over a corpus of real-shaped fixtures; regenerate with
  UPDATE_GOLDENS=1.

Second in the capability-layered series after the IR + loader; depends on that
PR and has no downstream dependencies of its own.
@libei

libei commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

cc @dmitrylychagin @amirhormati — PR2 of the semantic-model upstreaming series (BigQuery property-graph DDL generator). Stacked on PR1 (upstream-pr1-semantic-ir-loader); this diff is PR2-only (12 files). GitHub wouldn't let me add you as formal reviewers on the fork since you're not collaborators here — flagging for review.

Move the generator's behavior coverage onto readable, textual pairs: a
`<fixture>.yaml` semantic model next to a committed
`<fixture>.bigquery.golden.sql` showing the exact DDL and warnings. Adds
three fixtures to the e2e golden corpus:

- measure_lowering: every metric-to-MEASURE lowering shape (bare column,
  inline expression, shared operand reused once, COUNT(DISTINCT), an
  operand whose name collides with the metric, and a qualifier preserved
  inside a string literal).
- metric_skips: each unplaceable metric (spans multiple tables, ratio,
  unsupported aggregate, name collision) skipped and reported, with the
  one valid measure and its edge still emitted.
- keyless_dimension: a keyless node skipped and its edge omitted.

bigquery.test.ts now holds only what the open AI-first format cannot
express (it is a subset of the IR): an M:N association edge (hand-built
IR, checked against a committed golden file), the IR-contract cases the
loader never produces (COUNT(*) with a declared home; declared entities
disagreeing with the expression), degenerate inputs, and the structural
MEASURE-shape invariant guard run over the loaded fixtures. The two
inline VERIFIED_*_DDL strings are gone (now golden files).

No generator changes; behavior coverage is unchanged.
@libei
libei marked this pull request as ready for review August 2, 2026 06:37
@libei
libei merged commit 3f7a7cd into upstream-pr1-semantic-ir-loader Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant