diff --git a/GENERATION.md b/GENERATION.md new file mode 100644 index 00000000..a0e3b73b --- /dev/null +++ b/GENERATION.md @@ -0,0 +1,48 @@ +# MobilityDuck generation — the canonical per-binding generator policy + +This document is the contract for how MobilityDuck is generated, under the ecosystem-wide +per-binding generator policy. + +## The policy (ecosystem-wide) + +Every MobilityDB language/surface binding is a **pure projection of the MEOS-API catalog**, +and **each binding owns its own generator, in its own repo**, in a canonical layout. The +single source of truth is the **catalog** (`MEOS-API/output/meos-idl.json`, generated from +the MEOS C headers). A binding is an independent, plug-and-play module that owns its +generation. + +Each binding repo satisfies the same invariants: in-repo generator; own +`tools/pin/compose-order.txt`; vendored/pinned catalog; thin language projection +(language-neutral decisions live in the catalog); full automation toward a zero-hand-written +surface (generate-then-retire; the last green-CI version is the equivalence probe). + +## Current state and the canonical target + +MobilityDuck's UDF layer is **hand-written C++** today. The canonical target is to **generate +the DuckDB scalar UDFs from `meos-idl.json`** via an in-repo generator +(`tools/codegen_duck_udfs.py`), organized **by `@ingroup` group** (one registration unit per +group, the same structure as the MEOS reference manual and the JMEOS/Spark generators), +family-gated by `#ifdef MEOS_ENABLE_` emitted from catalog metadata. Marshalling +crosses MEOS values in-process as DuckDB `BLOB` (`BlobToTemporal` / `TemporalToBlob`, and the +per-family `BlobTo` helpers), with the per-thread MEOS-init guard asserted in every +emitted body. + +## Generate-then-retire — the green-CI version is the probe + +The hand-written UDFs are replaced **family by family, never wipe-first**: + +1. generate the full surface, build the extension green; +2. **prove generated ⊇ hand** against the **last green-CI version** (the equivalence probe) + — `scripts/parity-audit.py` + the full sqllogictest suite + the BerlinMOD benchmark; +3. retire the hand registrations for that family (drop the coexistence prefix); +4. repeat. End state: zero hand-registered scalar UDFs; only non-generatable hand code + survives, each justified (type registration, casts, aggregates, table functions). As the + generated surface lands, most of the hand feature PRs in `tools/pin/compose-order.txt` are + mooted. + +## Pinning + +The MEOS surface (vcpkg portfile + the vendored `meos-idl.json`) is pinned to a MobilityDB +`ecosystem-pin-*`. That pin is the *catalog/surface* input; MobilityDuck's own +`tools/pin/compose-order.txt` governs *this repo's* PR accumulate. See it for the composing +set and the disposition of every open PR. diff --git a/tools/pin/compose-order.txt b/tools/pin/compose-order.txt new file mode 100644 index 00000000..eb9030d8 --- /dev/null +++ b/tools/pin/compose-order.txt @@ -0,0 +1,93 @@ +# USER-APPROVED-PIN-WRITE — creating MobilityDuck's committed pin manifest (user 2026-06-25, +# per-binding generator policy rollout). New file in the MobilityDuck repo, NOT a mutation +# of MobilityDB's pin tooling. +# +# MobilityDuck pin — THE canonical, dependency-ordered fold manifest (per-binding policy). +# +# Derived from the LIVE base<-head DAG (gh pr list, verified this turn), organized into the +# established waves. Within a stack, the BASE folds before its HEAD. Edit THIS file (reviewed) +# — never re-derive ad hoc. (policy: generator-per-binding-canonical-policy, binding-compose-order-manifest) +# +# NOTE: MobilityDuck is today HAND-WRITTEN C++ UDFs (the 30 PRs below are hand features). The +# catalog-driven generator (`tools/codegen_duck_udfs.py`) is the planned migration — see +# GENERATION.md; it is NOT yet on GitHub. These hand PRs dissolve as generation lands +# (generate-then-retire). base = current origin/main. +# +# Format: # role. '?' = membership/order UNCONFIRMED. + +# ── WAVE 0 — PIN FOUNDATION (the MEOS pin bump everything rebases on) ── +197 fix/bump-meos-pin-clean # pin vcpkg MEOS to the canonical ecosystem SHA + meosType->MeosType + # (the clean pin bump; the `fix/bump-meos-pin` base branch of the stacks below) +190 fix/worker-thread-linkage-crashes # worker-thread + linkage crash fixes (head: fix/meos-error-longjmp / #149) +185 fix/stbox-from-hexwkb-nul-terminate # stboxFromHexWKB past-buffer read (#170) +173 fix/span-bins-bind-copy-unique-ptr-conv +188 fix/normalize-tnumber-mult-to-mul # mult -> mul naming +165 cleanup/retire-geodetic-stbox-workarounds + +# ── WAVE 1 — EXTENDED TYPES (port stack; fold bottom-up: tcbuffer<-tnpoint<-tpose<-trgeometry<-pointcloud<-tpcpoint<-tpcpatch) ── +150 feat/tnpoint-port-core # base feat/tcbuffer-port-core +151 feat/tpose-port-core # base feat/tnpoint-port-core +153 feat/trgeometry-port-core # base feat/tpose-port-core +154 feat/pointcloud-vcpkg-enabler # base feat/trgeometry-port-core +155 feat/tpcpoint-port-core # base feat/pointcloud-vcpkg-enabler +156 feat/tpcpatch-port-core # base feat/tpcpoint-port-core + +# ── WAVE 2 — PARITY (cell-index/json/trig stack: th3index<-h3-prefilter<-{setset,asmfjson,expandspace,trig}<-pin-11c<-tjsonb) ── +193 feat/parity-th3index # full H3 cell index API (base fix/bump-meos-pin) +198 feat/parity-h3-static-prefilter # base feat/parity-th3index +192 feat/setset-spatial-join-udfs # base feat/parity-h3-static-prefilter +194 feat/asmfjson-number-temporal-types # base feat/parity-h3-static-prefilter +195 feat/expandspace-tgeo-overloads # base feat/parity-h3-static-prefilter +199 feat/parity-tfloat-trig # base feat/parity-h3-static-prefilter +200 feat/parity-pin-11c # base feat/parity-tfloat-trig (a pin bump inside the parity stack) +201 feat/parity-tjsonb # base feat/parity-pin-11c +203 feat/parity-tquadbin # QUADBIN/TQUADBIN cell index (main-based; bumps pin) + +# ── WAVE 3 — GEOGRAPHY ── +168 doc/geography-boundary-design +169 feat/register-geography-logicaltype +174 feat/geography-io-udfs +175 feat/geography-casts +176 feat/geography-operations +177 feat/geography-test-matrix + +# ── WAVE 4 — FEATURES ── +205 feat/parity-raster-quadbin # rasterTileValueQuadbin + trajectoryQuadbins (Raquet sampling) + +# ── WAVE 5 — MULTI-VERSION / CI ── +166 feat/multi-duckdb-version-foundation +167 ci/multi-duckdb-version-matrix +186 build/fast-extension-iteration +189 ci/parallel-extension-build +170 ci/exclude-osx-arm64-pending-hex-wkb +171 ci/probe-mingw-build +172 ci/probe-musl-build +184 fix/icu-resilience-version-agnostic +182 fix/meos-tz-init-resilient +181 fix/tgeogpoint-test-tz-pinned + +# ── WAVE 6 — DIAGNOSTICS / FIXES ── +162 diag/hex-wkb-odd-length +180 feat/hex-wkb-diag +163 fix/drop-range-todos + +# ── WAVE 7 — DOCS ── +158 feat/edge-to-cloud-quickstart-rebased +159 doc/reviewer-guide-rebased +160 consolidate/pr-coordination-and-tz-lint-rebased +179 feat/iceberg-polaris-readiness +183 docs/duckdb-version-alignment-geoparquet + +# ── WAVE 8 — BENCHMARK (evidence vehicle; not a deliverable) ── +196 feat/berlinmod-canonical-queries +202 consolidate/berlinmod-canonical-queries + +# ── PIN BUMPS (pin mechanics, fold near WAVE 0 at assembly) ── +204 feat/pin-2026-06-18b # bump pinned MEOS to ecosystem-pin-2026-06-18b + +# ════════════════════════════════════════════════════════════════════════════════════ +# The deep parity/pin bumps inside the stacks (#200 pin-11c, #203/#204) are pin MECHANICS, not +# features; at assembly time the binding pins to the LATEST ecosystem pin once (GENERATION.md). +# The catalog generator (tools/codegen_duck_udfs.py) is the endgame: once generated ⊇ hand is +# proven, most of these hand feature PRs are mooted (generate-then-retire). +# ════════════════════════════════════════════════════════════════════════════════════ diff --git a/tools/regen-from-pin.sh b/tools/regen-from-pin.sh new file mode 100755 index 00000000..a44e5b24 --- /dev/null +++ b/tools/regen-from-pin.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# regen-from-pin.sh — regenerate the MobilityDuck UDFs from the MEOS catalog (per GENERATION.md). +# +# Usage: tools/regen-from-pin.sh +# env: CATALOG = path to meos-idl.json produced by MEOS-API run.py (required) +# MEOS_HEADERS = the installed MEOS headers dir for the pin-gate (required by the generator) +# +# NOTE: the catalog generator `tools/codegen_duck_udfs.py` is the target (GENERATION.md); it +# lands generate-then-retire alongside the hand-written UDFs. Until it is in the tree this +# script documents the invocation. Invoked standalone, or by tools/ecosystem-generate.sh. +set -euo pipefail +PIN="${1:?usage: regen-from-pin.sh }" +CATALOG="${CATALOG:?set CATALOG to the meos-idl.json from MEOS-API run.py}" +HERE="$(cd "$(dirname "$0")/.." && pwd)" +GEN="$HERE/tools/codegen_duck_udfs.py" + +if [ ! -f "$GEN" ]; then + echo "NOTE: $GEN not present yet (the catalog generator lands generate-then-retire — see GENERATION.md)." + echo " Once it is in the tree, this script regenerates src/generated/generated_temporal_udfs.cpp." + exit 0 +fi + +# generator CLI: codegen_duck_udfs.py +python3 "$GEN" "$CATALOG" "$HERE/src/generated/generated_temporal_udfs.cpp" "${MEOS_HEADERS:?set MEOS_HEADERS to the installed pin headers}" + +# build-verify the extension (the in-repo fast build target) +( cd "$HERE" && cmake --build build/release --target mobilityduck_loadable_extension ) \ + || echo "WARN: MobilityDuck extension build returned non-zero" +echo "[mobilityduck] regenerated UDFs from catalog at pin $PIN"