mdcode: add Semantic Model IR and model loader by libei · Pull Request #258 · GoogleCloudPlatform/knowledge-catalog

libei · 2026-08-02T05:39:07Z

First of a capability-layered series merging the semantic-model tooling from the libei/knowledge-catalog fork back into upstream. This PR is the foundation the rest of the series builds on; it has no downstream dependencies.

What this PR adds

A destination-agnostic Semantic Model IR and a loader for the Open Semantic Interchange (OSI) / Apache Ossie format.

src/libts/semantic/ir.ts — the in-memory contract: a graph of entities (nodes) and relationships (edges) with model-level metrics, AI-first annotations (ai_context), and verbatim vendor custom_extensions for round-trip fidelity. datatype is a closed, case-sensitive vocabulary mirroring Ossie's DataType.
src/libts/semantic/loader.ts — parses OSI YAML/JSON into the IR. zod schemas form the faithful parse layer; the IR is the normalized/lowered layer (dialect selection, physical-source FQN normalization, ai_context normalization, metric-entity inference). Fail-loud on off-vocabulary datatypes; warns (never silently drops) on soft issues such as missing primary keys or unplaceable metrics.
src/libts/semantic/sql_expr_utils.ts — literal-aware helpers for the entity-qualified SQL expressions used by metric-entity inference and qualifier stripping. Ignores text inside string literals and recognizes both bare and BigQuery backtick-quoted qualifiers.
Fixtures + tests — real OSI models (TPC-DS and a star schema), a multi-dialect model exercising expression selection across ANSI / Snowflake / Databricks, and models carrying vendor custom_extensions blocks (Databricks, DBT, Google, Salesforce). Fixtures are validated against the vendored OSI JSON Schema (via ajv), plus loader unit tests.
package.json — adds ajv (JSON Schema validation for the fixtures) and a test:semantic script wired into test; the package-lock.json churn is the transitive closure of that one dependency.

Scope

IR + loader only. Downstream consumers — BigQuery property-graph DDL generation, the CLI push path, and Knowledge Catalog emit/pull — follow in later PRs in the series.

Testing

npx tsc --noEmit — clean
bun test tests/libts/semantic/ — 67 pass / 0 fail

libei · 2026-08-02T05:39:29Z

@ChrisCChan @dmitrylychagin — requesting your review on this one. Fork permissions block a formal review request, so flagging you here. This is the first of a capability-layered series (IR + loader; downstream BigQuery/CLI/KC consumers follow in later PRs).

libei · 2026-08-02T05:41:26Z

@amirhormati — adding you as a reviewer on this one (fork permissions block a formal review request, so flagging you here). First of a capability-layered series: Semantic Model IR + OSI loader; downstream BigQuery/CLI/KC consumers follow in later PRs.

amirhormati · 2026-08-03T17:29:34Z

What is the plan to support properties on edges?

amirhormati · 2026-08-03T17:17:49Z

+            datatype: Decimal
+            expression:
+              dialects:
+                - dialect: ANSI_SQL


Should this be ANSO_SQL or GOOGLE_SQL? I know GOOGLE_SQL is not part of the standard but we should probably show out best case here?

Good point. Switched to BigQuery

Note each engine support GOOGLE SQL differently (different subsets). So we use the product name here.

libei · 2026-08-03T17:51:18Z

What is the plan to support properties on edges?

Good question. This PR mostly focusing on compatibility with OSI. I will add properties support in the BQ Graph publication PR (coming soon)

dlychagin-gg · 2026-08-03T21:58:02Z

+  name: string;
+  // Fully-qualified, normalized physical source: a dotted table reference
+  // (`project.dataset.table`, or a four-part Lakehouse catalog name
+  // `project.catalog.namespace.table`), or a verbatim query. The loader


nit. let' remove "or a verbatim query" for now

Done -- dropped the phrase from the comment. (The loader still passes a non-table source through unchanged if one shows up, but the IR contract no longer advertises it.)

dlychagin-gg · 2026-08-03T23:18:25Z

+  // `project.catalog.namespace.table`), or a verbatim query. The loader
+  // produces it: identifiers are unquoted and a missing project/dataset is
+  // filled from options; four-plus-part names are passed through untouched.
+  dataSource: string;


I think the value needs to be a resource id, not 3/4 part name. //bigquery.googleapis.com/projects/{PROJECT_ID}/datasets/{DATASET_ID}/tables/{TABLE_ID}

The IR is meant to stay destination-agnostic, so it holds the normalized project.dataset.table (or 4-part Lakehouse name). The //bigquery.googleapis.com/... resource id is a BigQuery/Dataplex-specific encoding -- I'd rather produce it at the push/emit layer that already knows the target than bake a BigQuery URI into the shared IR. Keeping the dotted form here for PR1; happy to revisit if you'd prefer the resource id live in the IR itself.

Reworked the comment so it no longer prescribes a 3/4-part shape. The IR now treats dataSource as an opaque, fully-qualified identifier -- it fixes neither a syntax (separators, number of parts, quoting) nor a naming scheme; that's whatever the source system uses. The BigQuery resource-id form (//bigquery.googleapis.com/projects/{PROJECT_ID}/datasets/{DATASET_ID}/tables/{TABLE_ID}) is a target-specific encoding, so I'd produce it in the push/emit layer that already knows it's targeting BigQuery rather than bake a BigQuery URI into the shared, destination-agnostic IR. Updated in the latest push.

dlychagin-gg · 2026-08-03T23:24:21Z

+  // that node; several: a cross-entity metric whose join path consumers resolve
+  // via the model's relationships. It MAY be empty when the expression names no
+  // entity (e.g. `COUNT(*)`); such a metric is not placeable and the loader warns.
+  entities: string[];


why do we have an array here? I believe, it should be just one entity

A metric expression can reference columns from more than one entity (e.g. across a join), so the list is the general form. When it references a single entity, that entity is the attach point and the expression doesn't need to repeat the qualifier; when it spans several, the expression has to name each entity inline anyway. So the array covers both cases -- single-entity is just the common one. Kept as an array.

Went with your instinct after another look -- changed it to a singular optional entity?: string (dropped the array in the latest push). The loader sets it only when the expression references exactly one entity (the node the metric attaches to). A cross-entity metric leaves it undefined and keeps its qualifiers inline in the expression; a COUNT(*)-style metric that names no entity also leaves it undefined (with the existing "references no known entity" warning).

dlychagin-gg · 2026-08-03T23:33:00Z

+    warnings.push(`${ctx}: 'to' dataset '${r.to}' is not defined in the model`);
+  }
+  if (r.from_columns.length !== r.to_columns.length) {
+    warnings.push(


why is this a warning? should it be an error instead?

Agreed -- now a hard error. An endpoint that isn't declared in the model throws instead of warning.

dlychagin-gg · 2026-08-03T23:34:07Z

+    warnings.push(`${ctx}: 'from' dataset '${r.from}' is not defined in the model`);
+  }
+  if (!keysByEntity.has(r.to)) {
+    warnings.push(`${ctx}: 'to' dataset '${r.to}' is not defined in the model`);


should it be an error?

Yes -- converted to a hard error too. A structurally invalid edge shouldn't pass silently.

dlychagin-gg · 2026-08-03T23:37:55Z

+                            warnings: string[]): Relationship {
+  const ctx = `relationship '${r.name}'`;
+  const fromKeys = keysByEntity.get(r.from);
+  if (fromKeys === undefined) {


we should require that fromColumns are defined. It should be an error if fromColumns are missing or empty array. fromKeys are not used in relationships.

Done. from_columns/to_columns are now required (zod .min(1)), and mismatched arity is a hard error. fromKeys is gone from relationships -- the source entity's own key is looked up from the entity, not duplicated on the edge.

dlychagin-gg · 2026-08-03T23:40:06Z

+  name: string;
+  source: RelationshipEnd;
+  destination: RelationshipEnd;
+  dataSource?: string;      // association/edge table (FQN); absent => direct FK join


dataSource and keys fields are never set by OSI at this point. should we remove them for now?

Done -- removed dataSource, keys (and fields) from Relationship. It's now a pure direct-FK edge: source/destination + their join columns. Edge-table / M:N support will return in the BigQuery-graph PR, where the generator actually consumes it and I can model the endpoints properly.

dlychagin-gg · 2026-08-03T23:45:42Z

+  }
+
+  // Fall back to the FK columns when the from dataset declares no primary key.
+  const sourceKey = fromKeys && fromKeys.length ? fromKeys : r.from_columns;


this seems confusing. we need to decide whether we need sourceKey at all or not. OSI model does not use it.

Agreed -- removed. The relationship no longer carries a source key; source/destination each hold only their join columns. A consumer that needs the source entity's PK looks it up from the entity.

dlychagin-gg · 2026-08-03T23:46:17Z

+    name: r.name,
+    source: {
+      entity: r.from,
+      joinKeys: { relationshipColumns: sourceKey, entityColumns: sourceKey },


I don't think joinKeys is needed here.

Done -- joinKeys is gone. RelationshipEnd is now just { entity, columns }.

Introduce the semantic-model subsystem foundation for the mdcode toolbox: - ir.ts: the Semantic Model IR contract (M0) — a pure, destination- agnostic representation of tables, dimensions, measures, joins and graph structure. - expr.ts: shared expression helpers used across the IR front-end. - loader.ts: parse AI-first semantic-model files (YAML) into the IR, validated with zod. - tests/libts/semantic/loader.test.ts: behaviour-driven loader tests. - package.json: add a `test:semantic` script and run it under `test`. This is additive and self-contained; no existing files change behaviour. Downstream emitters (BigQuery DDL, Knowledge Catalog) build on this IR in follow-up changes.

libei mentioned this pull request Aug 2, 2026

mdcode: generate BigQuery property-graph DDL from the Semantic Model IR libei/knowledge-catalog#24

Merged

libei force-pushed the upstream-pr1-semantic-ir-loader branch from 3f7a7cd to b34ae38 Compare August 2, 2026 06:47

libei marked this pull request as ready for review August 2, 2026 06:47

amirhormati self-requested a review August 3, 2026 17:29

amirhormati reviewed Aug 3, 2026

View reviewed changes

libei force-pushed the upstream-pr1-semantic-ir-loader branch from b34ae38 to d015f89 Compare August 3, 2026 18:00

amirhormati approved these changes Aug 3, 2026

View reviewed changes

dlychagin-gg reviewed Aug 3, 2026

View reviewed changes

libei force-pushed the upstream-pr1-semantic-ir-loader branch 3 times, most recently from afe1aa1 to f799a42 Compare August 4, 2026 02:56

libei force-pushed the upstream-pr1-semantic-ir-loader branch from f799a42 to 2259c10 Compare August 4, 2026 02:58

dlychagin-gg approved these changes Aug 4, 2026

View reviewed changes

libei merged commit 599a240 into GoogleCloudPlatform:main Aug 4, 2026
6 checks passed

This was referenced Aug 4, 2026

mdcode: generate BigQuery property-graph DDL from the Semantic Model IR #269

Merged

mdcode: add semantic-model scope and BigQuery Graph push #270

Merged

Uh oh!

Conversation

libei commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR adds

Scope

Testing

Uh oh!

libei commented Aug 2, 2026

Uh oh!

libei commented Aug 2, 2026

Uh oh!

amirhormati commented Aug 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

libei commented Aug 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

libei commented Aug 2, 2026 •

edited

Loading