Generate the temporal FromHexWKB and FromMFJSON parsers#212
Open
estebanzimanyi wants to merge 83 commits into
Open
Generate the temporal FromHexWKB and FromMFJSON parsers#212estebanzimanyi wants to merge 83 commits into
estebanzimanyi wants to merge 83 commits into
Conversation
Register the quadbin / tquadbin cell-index types and their cast, accessor, and operator surface, with the rtree index module and build wiring.
… 1.4.4 DuckDB runs scalar, cast and table-function bodies on worker threads whose MEOS thread-local state is uninitialised; initialize MEOS once per worker thread at the registration chokepoints and table-function Init bodies, and consume the DuckDB 1.4.4 unique_ptr_cast Copy signature.
The restrict argument was dropped from the spatiotemporal relationships in the 1.4 orthogonalization; drop the non-canonical 3-arg overloads and use the 2-arg form in the tests.
Refresh expecteds that drifted with the pin: share-inclusive span adjacency, ln/log10/exp densification, and right-open step segment-duration bounds, all matching MobilityDB canonical output.
Pin the MEOS vcpkg port to ecosystem-pin-2026-06-22a (043d5e72), carrying the GEOS CMake CONFIG-target fix for the macOS build. Consume the pgtypes public leaf headers for date/timestamp arithmetic and drop the obsolete source patches the clean MEOS-only surface makes unnecessary.
The MEOS type-name catalog (meos/src/temporal/meos_catalog.c) defines the canonical type names in lowercase (tint, intspan, stbox, tgeompoint, ...). Name the DuckDB type accessors and SQL aliases to match that canonical for the temporal, span, box, and temporal-geo families, consistent with the set and spanset accessors that already use it. The third-party duckdb-spatial GEOMETRY type is left unchanged.
The MEOS catalog provides the tbigint type (T_TBIGINT, tbigint_in/out) in the pinned library. Register it in the binding the same way as tint: a BASE_TYPES entry, a DEFINE_TEMPORAL_TYPE accessor, a RegisterType call, membership in AllTypes(), and an alias-to-meosType catalog entry. This wires tbigint into the generic temporal cast and function loops and the catalog-driven generator.
Add a catalog-driven generator (tools/codegen_duck_udfs.py) that emits DuckDB scalar UDFs from the vendored MEOS-API catalog (tools/catalog/meos-idl-22a.json), organized by doxygen @InGroup group into one RegisterGenerated_<group> function each, mirroring the structure of the other ecosystem binding generators. The generated surface (src/generated/generated_temporal_udfs.cpp) is wired into the extension sources and registered last in LoadInternal, after every type accessor exists. Generated names carry a coexistence prefix so they run alongside the hand-written surface; a function is gated against the installed MEOS headers so only declared symbols are emitted.
…t guard MEOS keeps locale and collation, the session timezone, the PROJ context, and random-number state in thread-local storage, so every generated UDF body must run EnsureMeosThreadInitialized() before any MEOS call. After writing the output, scan every emitted body and exit non-zero if the guard is missing, so a future emit path cannot silently drop it.
…rations The frechetDistance, dynTimeWarpDistance, and hausdorffDistance functions are emitted by the catalog-driven generator (group meos_temporal_analytics_similarity). Add that group to RETIRED_GROUPS so the generated surface owns the canonical names (no coexistence prefix), and remove the corresponding hand registrations; the discreteFrechet and dynTimeWarp aliases and the *Path variants, which the generator does not emit, are kept.
Two ways a RETIRED_GROUPS retirement can silently lose coverage, now caught at generation time by reading the actual emitted output: - Trap 1 (split name): a name emitted both bare and g_-prefixed means a retired group and a non-retired group share it, so a query resolves only one spelling. Shared bare/operator dialect must be retired as a coherent wave. - Trap 2 (uncovered): every @sqlfn of a retired group must be emitted, or retiring it drops the function. A documented generator-shape gap kept by hand is allowlisted in RETIRE_UNCOVERED_OK with a reason; anything else is fatal. The checks are output-neutral (read-only over the generated file) and mirror the existing per-thread-init guard. RETIRE_UNCOVERED_OK currently holds dynTimeWarpPath/frechetDistancePath (LIST(STRUCT) path returns the generator cannot yet emit).
The vcpkg meos port linked only a partial family set (H3 off; cbuffer, pose, rgeo, pointcloud and the Arrow export absent), so the standalone libmeos the extension loads was a strict subset of the catalog surface the generator targets, and a faithful regen emitted unlinkable symbols. Enable every family at the pin (which already folds all family PRs and carries the MEOS_OBJECTS shared-link fix, proven by quadbin linking): CBUFFER, H3, POSE, POINTCLOUD and ARROW alongside JSON, NPOINT, QUADBIN. RGEO follows POSE automatically; RASTER stays off (PostgreSQL-only). H3 uses the system libh3 (auto-found); Arrow is header-only (vendored arrow and nanoarrow headers). POINTCLOUD links pointcloud-pg/lib/libpc.a, a build artifact not shipped in the tarball, so build it first from the vendored autotools tree (autogen, configure, make -C lib; the PGXS pgsql subdir is skipped). Port-version bumped so vcpkg rebuilds the port. The libmeos now loads with every family plus the 18 Arrow roundtrip helpers, a faithful regen from the catalog has zero of its 858 called symbols missing, and the release suite passes (1331 assertions, 60 cases).
With the vendored libmeos now built with every family, the generator emits the cross-family conversion UDFs that were previously dropped (tcbuffer_to_tfloat, ttext_to_tjsonb, tbigint_to_tquadbin, the h3/npoint conversions, ...). Make the generated registrations a faithful, buildable projection of the catalog again: - Regenerate src/generated/generated_temporal_udfs.cpp: the conversion registrations return; the file is once more a clean regen of the catalog. - meos_wrapper_simple.hpp: include the per-family public headers (alphabetical) so the family type and conversion declarations are visible to the generated translation unit; they live in the family headers, not the meos.h umbrella. - CMakeLists.txt: add the libh3 include path (meos_h3.h includes <h3api.h>), discovered with NO_CMAKE_FIND_ROOT_PATH since the vcpkg toolchain confines find_path to the vcpkg root and never sees the system /usr/include/h3.
…talog
The catalog carries function-shape metadata (shape.arrayReturn, from the
MEOS-API shapeinfer pass) plus the struct layouts (warp/Match = {i, j}).
Use it to generate the SETOF-returning similarity path functions instead
of hand-writing them:
- codegen: a new array-return shape (poc_path + emit_path_table) emits a
DuckDB TABLE function (bind/init/exec) whose output schema and per-row
marshalling are driven entirely by the catalog struct fields. The
registration goes through loader.RegisterFunction(TableFunction(...)),
registered over every temporal type. The retire-safety guard now counts
TableFunction registrations too.
- vendored catalog: enriched with the per-function shape and the struct
table (same 4465-function surface, shape added).
- retire the hand surface: the SimilarityPath table functions, the scalar
RunSimilarityPath path executors, and the non-canonical discreteFrechet /
dynTimeWarp aliases (MobilityDB-canonical names are frechetDistance /
dynTimeWarpDistance, both generated). frechetDistancePath / dynTimeWarpPath
RETURNS SETOF warp, so the canonical form is the table function.
- tests: the path tests use the canonical SETOF/table form
(SELECT i, j FROM ...Path(...)); 038 uses the canonical dynTimeWarpDistance.
Release suite: 1339 assertions in 60 cases pass.
A MEOS span/box function that can produce an empty result returns a NULL pointer (e.g. intersection_span_span on disjoint spans). The generated pointer-returning bodies marshalled that pointer straight through SpanToBlob, which reads sizeof(Span) from it -> a NULL dereference and a crash on the empty case (latent while the names carried the coexist prefix and were not exercised by the suite). Make the generator emit null-safe bodies for the pointer-returning span shapes (unary X->X and binary X,X->X, shared by span/spanset/stbox/tbox): use ExecuteWithNulls and, when MEOS returns NULL, set the row invalid and return an empty string_t -- mirroring the hand SpanFunctions handlers. Proven by exercising the generated function directly: the disjoint-span intersection now returns SQL NULL, the overlapping case returns the intersection. Release suite: 1339 assertions in 60 cases pass.
Extends the span/box null-safety (previous commit) to the rest of the generated scalar surface: any MEOS function that returns a NULL pointer for an empty/undefined result (set intersection/minus on disjoint inputs, temporal at/minus restrictions with no match, etc.) now maps to SQL NULL instead of dereferencing NULL in *ToBlob and crashing. Add centralized null-aware marshalling helpers TemporalToBlobN / SetToBlobN (if the pointer is NULL, set the row invalid via the mask and return an empty string_t) and route the pointer-returning emitter shapes through them with ExecuteWithNulls: emit_set (unary/binary set), emit_body, emit_body_binary, emit_body_ternary, emit_binary_tt, emit_scalar_first. Mirrors the hand binding's ExecuteWithNulls handlers; bool/scalar returns keep the simple Execute path. Proven by exercising the generated functions directly: set intersection on disjoint sets returns SQL NULL, the overlapping case returns the result. Release suite: 1339 assertions in 60 cases pass.
Drop the g_ coexist prefix so the generated UDFs own the bare canonical SQL names, and correct the generated geo return types: ever/always spatial relationships (group *_rel_ever) return BOOLEAN, temporal spatial relationships (group *_rel_temp) return tbool, the temporal distance tDistance returns tfloat, and geo conversions (X_to_t{geometry,geography,geompoint,geogpoint}) return the target type.
Scan the extended-family headers (cbuffer, h3, json, npoint, pointcloud, pose, quadbin, rgeo) in the pin/ABI gate, matching the per-family headers the generated translation unit includes, so the cross-family conversions are emitted.
Drop the hand tint value-accessor, trend, and temporal-distance registrations that the generated surface now owns with the canonical return types.
…ames 030_temporal_compops called the retired hand names temporal_teq/tne/tlt/tle/tgt/tge; the comparison surface is now generated as tEq/tNe/tLt/tLe/tGt/tGe (RETIRED_GROUPS meos_temporal_comp_temp). Update the test to the generated names.
…he composed catalog Vendor the complete MEOS-API catalog (4492 fns; the composed all-open-PR surface) and bump the pinned MEOS base to upstream master bb5f9e70 (all families + Arrow/Raster), regenerating the scalar UDFs. PIN and portfile REF both bb5f9e70; vcpkg port-version 7. Note: bb5f9e70 trails current upstream master (b529843ae8) by the arc-clip + cmake-ALL commits; rebasing the base to current master is a follow-up.
The pointcloud-pg autotools configure step fails to build on arm64-linux, and the binding surfaces no pointcloud type or function. Drop the pointcloud-pg build and the POINTCLOUD option from the meos port, and the meos_pointcloud.h include from the family header block.
…neration - Drop the g_ coexistence prefix for good (remove COEXIST_PREFIX and the --prefix flag); generated UDFs always register under the canonical @sqlfn name. Hand code is retired via RETIRED_GROUPS (generate-then-retire), never a transition prefix. - Add geometry-argument marshalling (poc_geo_temporal/emit_geo_temporal): generate the geo spatial-relationship surface (eContains/eIntersects/ eDwithin/tIntersects/tDwithin/... over tgeo x geometry, both argument orders) via GeometryToGSerialized(blob, tspatial_srid(t)); ever/always int -> nullable BOOLEAN, temporal -> tbool. - Restrict conversion UDFs to registered type families (REGISTERED_FAMILIES) so no UDF is emitted for an unregistered family. - Generate temporal_lcss_distance via the (Temporal, Temporal, scalar) shape. Build + full test suite green (1339 assertions / 60 cases).
Per the per-binding-canonical generator policy, each binding owns the executable form of its GENERATION.md. The script bumps the vcpkg MEOS portfile REF + SHA512 and vcpkg.json port-version, writes tools/catalog/PIN, vendors the catalog produced by MEOS-API run.py, and (given the freshly installed headers) runs the generator pin-gated and builds the extension.
- portfile REF -> b71198726a (current upstream master), new SHA512, port-version 10; PIN + vendored catalog meos-idl-b71198726a.json (4519 functions / 2555 @sqlfn), replacing the stale meos-idl-26a.json. - Enable RASTER=ON: the raquet tile type (T_RAQUET, MobilityDB #1332) is a GDAL-free varlena value type that links standalone (15 raquet symbols in libmeos), so the standalone MEOS surface matches the full catalog. - Regenerate src/generated/generated_temporal_udfs.cpp (pure add-only, +36 registrations): picks up the completed geo-first spatial-relationship grid (eDisjoint/eIntersects/eTouches/eDwithin/tDwithin with geometry as the first argument), now that the MEOS API exposes those symbols. Full test suite green (1339 assertions in 60 test cases).
reg_scope maps the abstract tpoint_ supertype to its two point subtypes, and ret_temporal_type takes the output temporal type from the catalog's single unambiguous SQL return subtype (falling back to the name heuristics only for input-polymorphic returns, whose result preserves the input type). This generates getX/getY/getZ, azimuth, speed, cumulativeLength, angularDifference, isSimple, length and the atGeometry/atValue/minusGeometry/ minusValue restrictions on tgeompoint and tgeogpoint, and gives trend a tint result and derivative a tfloat result, matching the MobilityDB SQL surface. Full test suite green (1339 assertions in 60 test cases).
…talog reg_scope maps the abstract tspatial_ supertype to all four geo temporal types. This generates setSRID, transform and transformPipeline (each preserving the operand type) and asText/asEWKT (returning text) on tgeometry, tgeography, tgeompoint and tgeogpoint. Full test suite green (1339 assertions in 60 test cases).
Pins are abandoned: the vcpkg meos portfile REF is a raw upstream MobilityDB/MobilityDB commit that tracks master. Bump REF + SHA512 to master 4bf009b1 (port-version 11) and vendor the catalog regenerated from that checkout (meos-idl-4bf009b120.json, 4526 functions / 2555 @sqlfn). Remove the legacy pin-model tooling — tools/catalog/PIN and tools/regen-from-pin.sh — now that freshness is the base REF versus upstream master. Full test suite green (1339 assertions in 60 test cases).
The bounding-box topological operators/functions (~=/@>/<@/&&/-|- and same_bbox/contains_bbox/contained_bbox/overlaps_bbox/adjacent_bbox) have type-dependent semantics: the generic same_temporal_temporal compares only the time box, same_tnumber_tnumber the value+time (tbox), and same_tspatial_tspatial the space+time (stbox). The generator scoped the generic *_temporal_temporal backing to every temporal type from its `(Temporal *, Temporal *)` C signature, so it was also registered for the number and spatial types. Because all MEOS types are alias-tagged BLOBs and DuckDB overload resolution is last-registration- wins, that generic time-only op shadowed the correct value-/space-aware backing: `tgeompoint ~= tgeompoint` and `tbigint ~= tbigint` silently compared only the time dimension (returning true for differing space/value at the same instant). Scope each backing to the types whose bounding box it models: the generic temporal_temporal to tbool/ttext (tstzspan), tnumber_tnumber to tint/tfloat/ tbigint (tbox, all tnumber_type), leaving the spatial types to the already generated tspatial_tspatial (stbox). Add a regression test asserting the full box dimension is compared for tspatial, tnumber (incl. tbigint) and tbool.
Bump the vcpkg meos port (REF + SHA512, port-version 4) to the current MobilityDB master and re-vendor tools/catalog/meos-idl.json regenerated from it. The refreshed catalog carries the array-return shape metadata now resolved at the source: element type on all 165 array-returning accessors and the trailing int* count length parameter on set_spans, spanset_spans, spanset_spanarr, spanset_sps and the jsonb/pointcloud getValues accessors. It also picks up the upstream fix that tags ttext_initcap with its own @csqlfn, so the generated temporal text function is initcap instead of the mistagged lower.
…and implementations Drive the trailing-count array-return convention (shape.arrayReturn with element + lengthFrom.name=count) through codegen_duck_udfs.py so the LIST-returning accessors are emitted directly from meos-idl.json, gated on each overload's canonical sqlSignatures return type: getValues(<set>) / getValues(tbool) -> LIST(<element>) timestamps(<temporal>) (9 types) -> LIST(TIMESTAMP_TZ) spans(<set> | <spanset> | <temporal>) -> LIST(<span type>) tboxes(tint | tfloat) -> LIST(tbox) Element marshalling picks the DuckDB child type from the catalog element (scalars, epoch-corrected TIMESTAMP_TZ/DATE, text, and Span/STBox/TBox value structs as BLOB-backed named LIST types) with no name heuristics, and the SQL element type comes from the per-overload sqlSignatures so tint/ tfloat/ttext getValues correctly stay as their spanset/set forms rather than being mis-emitted as LIST. Retire the now-superseded hand registrations and their implementations: Set_values, Set_spans, Spanset_spans, Temporal_timestamps (incl. the five geo-family regs) and Tnumber_tboxes. The generated accessors are the sole implementation; the full suite passes (1372 assertions / 62 cases).
…al ports The meos vcpkg port enables H3=ON (find_library h3 + find_path h3api.h) and RASTER=ON (find_package GDAL), but declared neither dependency, so it relied on a system libh3-dev / libgdal-dev. The DuckDB extension CI builds each triplet inside a manylinux container that ships neither, so the meos configure aborted on arm64 with "H3=ON was requested but libh3 was not found" (the x64 job only appeared to pass because a developer's host happens to have libh3 installed). Declare h3 and gdal as vcpkg dependencies of the meos port so both families are provisioned by vcpkg on every triplet (x64/arm64/osx/wasm) — the cross-arch equivalent of the provision-meos action's apt libh3-dev / libgdal-dev. This is the "all families except POINTCLOUD" tier: POINTCLOUD FORCE-requires the vendored pgPointCloud PGXS build (a live pg_config) which has no vcpkg port, so it stays off until a vcpkg pointcloud port lets the port flip to true -DALL.
Enable every optional MEOS family in the vcpkg meos port via -DALL=ON, replacing the explicit family list that held POINTCLOUD OFF. POINTCLOUD now builds without a live pg_config: its pgPointCloud core (libpc) is compiled directly from lib/*.c as a CMake static library (MobilityDB #1370, on master 79f24b7948), so -DALL builds in the vcpkg/manylinux model with no system PostgreSQL. Its only new external dependencies are libxml2 and zlib, added to the port's vcpkg.json alongside h3 (H3) and gdal (RASTER); port-version bumped to 6. Bump the port REF/SHA512 to master 79f24b7948 and re-vendor tools/catalog/meos-idl.json from the same commit so the catalog stays a subset of the installed libmeos surface (portfile REF == catalog pin). The generated UDF surface is unchanged and the suite is green (1372 assertions in 62 test cases).
Retire the hand-registered timeSpan/valueSpan/tbox and tint/tfloat temporal-conversion scalar functions (@InGroup meos_temporal_conversion) in favour of catalog-driven generation. A new sqlSignatures-driven shape in tools/codegen_duck_udfs.py emits the unary Temporal -> Span/SpanSet/TBox/STBox conversions (temporal_to_tstzspan -> timeSpan, tnumber_to_span -> valueSpan, tnumber_to_tbox -> tbox); each overload's container return type comes straight from the catalog sqlSignatures (tint -> intspan, tbigint -> bigintspan, tfloat -> floatspan), replacing the previous name heuristic. The group is added to RETIRED_GROUPS and the retire-safety check verifies every @sqlfn of the group is generated. The generated surface also adds the tbigint overloads the hand layer was missing. The temporal-number cast helpers and their cast registrations are unchanged; only the scalar function registrations are retired. Full suite green (1372 assertions / 62 cases).
…enerated surface
The before/after/overbefore/overafter (time axis) and left/right/overleft/
overright (numeric axis on tnumber x {numspan, tbox, tnumber}) positional
predicates, plus the tspatial x {stbox, tspatial} spatial-axis variants, are
already emitted from the catalog sqlSignatures in the generated UDF file,
including the <<, >>, &<, &> operator forms. Delete the three redundant hand
registration blocks in temporal.cpp.
The generated surface follows the catalog exactly: positional overloads are
same-base-family only (before_temporal_temporal is tbool/ttext, numbers go
through _tnumber_tnumber, spatial through _tspatial_tspatial), so the old hand
loop's spurious mixed pairs (e.g. before(tint, tfloat)) are dropped, aligning
the DuckDB surface with MobilityDB. The non-canonical temporal_* snake aliases
are dropped in favour of the bare names.
Migrate the tests that used the retired snake aliases to the bare names
(034_temporal_posops, 040/041/042 parity).
…generated surface The before/after/overbefore/overafter time-axis predicates on tgeometry, tgeography and tgeogpoint against tstzspan (both directions) are already emitted from the catalog sqlSignatures in the generated UDF file — their backing MEOS exports (before_temporal_tstzspan etc., @InGroup meos_temporal_bbox_pos) carry the bare @sqlfn names and dispatch generically over every temporal subtype. Delete the redundant TIME_POS_REG macro blocks in the three geo ops files. These only ever registered the non-canonical temporal_* snake aliases; the bare names come from the generated surface. This removes the last hand-registered snake positional aliases in the binding.
Bump the meos vcpkg port REF to the current MobilityDB master (5c5b3cd25b) and re-vendor tools/catalog/meos-idl.json from the same SHA (portfile REF == catalog pin). -DALL stays on, so libmeos carries every family. This pulls in the time-axis span/set positional tag fix (before/after/overbefore/ overafter on tstzspan/dateset now resolve to their own name/operator instead of the value name left/right), so the generated surface emits before(tstzspan, ...) rather than left(tstzspan, ...); and the pointcloud gated-declaration catalog entries now captured by the header parser. Generated code regenerates byte-identical against the fresh installed headers; suite green (1372 assertions / 62 cases).
…rations The set/span/spanset relative-position operators (left/right/overleft/overright + the time-axis before/after/overbefore/overafter, with the <<//>>/&</&> operators) are now fully generated from the catalog. Close the one remaining shape gap: the mixed span<->spanset pair (left_span_spanset / left_spanset_span, ...), which the same-container (X,X) case skipped because the two operands are different containers. A new shape marshals each operand as its own container and registers the concrete type pairs (intspan x intspanset, ...) from the catalog sqlSignatures. Add meos_setspan_pos to RETIRED_GROUPS (retire-safety now covers every @sqlfn of the group) and delete the hand span_left/set_left snake registrations and their operator forms in span.cpp/set.cpp. duckdb_functions() shows 0 snake positional names remain, the bare left/right/before/after surface intact, and the mixed span<->spanset overloads present; suite green (1372 assertions / 62 cases).
…trations The set/span/spanset topological operators (contains/contained/overlaps/ adjacent and the @>/<@/&&/-|- operators) are generated from the catalog span and set shapes, under their bare portable names in both argument orders, including the symmetric value-first adjacent(value, span). The hand set_contains/span_contains snake-case functions and operator registrations are deleted; meos_setspan_topo joins RETIRED_GROUPS so the retire-safety ledger fails the build if the catalog ever drops coverage. The value-first adjacent(value, span) overload is backed by the adjacent_<type>_span MEOS-C functions, so the catalog is re-vendored and the MEOS port advanced to pick them up.
The meos port pulls h3 as a dependency (via -DALL → H3). The h3 v4.3.0 port at the builtin-baseline builds with ENABLE_LINTING ON, which sets the h3 target's C_CLANG_TIDY property. Under the wasm32-emscripten toolchain the runner's clang-tidy then runs on h3's own sources with warnings-as-errors and the vendored h3 code trips readability-braces-around-statements and non-const-global findings, failing the DuckDB-Wasm build. Add a version-identical h3 overlay port (same v4.3.0 REF and SHA512) that passes -DENABLE_DOCS=OFF -DENABLE_FORMAT=OFF -DENABLE_LINTING=OFF, so the build no longer lints upstream h3 sources. Newer upstream vcpkg h3 ports carry these same flags. This mirrors the existing gdal/proj/sqlite3 overlays under vcpkg_ports.
The temporal bounding-box topological operators (contains/contained/
overlaps/same/adjacent and @> <@ && ~= -|-) for temporal × temporal,
temporal × tstzspan and tnumber × {numspan, tbox} are generated from the
catalog sqlSignatures. Delete the hand registrations in temporal.cpp,
which duplicated the generated bare surface and additionally exposed the
non-canonical temporal_* snake aliases and a spurious mixed-family
cross-product, and add meos_temporal_bbox_topo to RETIRED_GROUPS.
Migrate the affected parity tests to the canonical bare names. Adjacent
uses TBox (value × time) adjacency, which holds when the operands are
adjacent in at least one dimension; the deleted hand temporal × temporal
handler computed time-only adjacency, so adjacent on two tints whose
integer value spans share a boundary is now correctly true.
Mirror MobilityDB's tools/codegen/inherited/ for DuckDB: a manifest, a per-behaviour template and a driver that emit the DuckDB scalar-function registrations every Temporal<T> subtype inherits (topological, as the first behaviour) — the same inherited surface PostgreSQL generates from its .sql templates, with only the output form changed. The live DuckDB surface is generated by tools/codegen_duck_udfs.py from the MEOS-API catalog; this generator is a design reference for the template-driven approach, kept honest by --validate, which proves it reproduces the catalog-generated topological surface byte-for-byte.
Add the circular-buffer temporal family to the DuckDB binding: the cbuffer value type and the tcbuffer temporal type, plus their catalog-generated operation surface. The type + text-I/O boundary is hand-registered in src/cbuffer/tcbuffer.cpp (cbuffer and tcbuffer are surfaced as BLOB with the canonical alias; the four VARCHAR in/out casts marshal through cbuffer_in/out and tcbuffer_in / temporal_out). Everything else is generated from the MEOS-API catalog by tools/codegen_duck_udfs.py: registering the family adds the Cbuffer boundary marshaller (a 4-byte-header varlena) and resolves the tcbuffer_* / _tcbuffer names to the tcbuffer type, yielding 77 functions — accessors, conversions (tfloat radius, tgeompoint centre), temporal distance, and the ever/always/ temporal spatial relationships. tcbuffer_make is the one function left ungenerated: its C signature is two polymorphic Temporal* operands (a tgeompoint and a tfloat) that the catalog cannot yet type per argument, so the generator excludes it (hetero-temporal- args) rather than register wrong argument types; it stays reachable through the text-I/O cast.
tcbuffer is a TSpatial<T> subtype, so it inherits the spatial behaviour previously confined to the geometry/geography temporal types, as well as the generic Temporal<T> behaviour of every temporal type. The first registration scoped tcbuffer only through its own tcbuffer_* / _tcbuffer names, which gave it the native operations but not the inherited surface: asText/asEWKT/SRID/ setSRID/transform, the bounding-box topological and positional operators, and the generic temporal duration/atTime/minusTime/deleteTime were all missing (a few appeared to work only through incidental BLOB-alias coercion). Introduce SPATIAL_ALLTYPES (the geo types plus tcbuffer, extensible to the other spatial families) and use it wherever the surface is "all spatial subtypes": the abstract tspatial_* scope, the generic Temporal<T> registration loop, and the time-restriction type set (ALL_TEMPORAL_ACCS). The geo supertype tgeo stays geometry+geography only. Also generate the shorter-arity overload of scalar-returning functions with a trailing default (duration(temporal), asText/asEWKT(tspatial)), which was previously hand-written for the geo types only. tcbuffer now inherits the full surface (asText, asEWKT, SRID, atTime with interpolation, the topological operators, etc.). This is the pattern to replicate to the remaining spatial families.
startValue and endValue return the family base value through the per-type
MEOS symbol (tcbuffer_start_value / tcbuffer_end_value -> Cbuffer *).
The generator gains a base-value pointer-return shape in shape_emittable:
BASEVAL_PTR_RET is the set of PTR_RET entries that are neither a temporal
handle nor a container (currently Cbuffer). The owned pointer marshals
through {Base}ToBlobN, and reg_scope keys the per-type symbol to its own
family, so startValue/endValue register on tcbuffer returning cbuffer.
Add the GSERIALIZED-return shape to the generator: an owned geometry return marshals to the DuckDB GEOMETRY type via GSerializedToGeometry (the inverse of GeometryToGSerialized). The shape is scoped to the non-geo spatial families so it does not double-register against the hand-written geometry accessors on the geo temporal types. For tcbuffer this generates convexHull(tcbuffer) -> geometry. convexHull is a linear (GEOS-polygonized) hull of the traversed area, not a curved geometry; the test uses a zero-radius input for a stable, arc-free result.
The committed tools/catalog/meos-idl.json was a point-in-time snapshot that drifts from the libmeos the extension is built against. Drop it and derive the catalog in CI from MobilityDB via the shared MEOS-API provision-meos action — the same model the JVM bindings (JMEOS/Spark/Flink/Kafka) use. A new generate workflow reads the MobilityDB commit from the vcpkg MEOS port REF (the single source of truth for which MobilityDB the port builds), derives meos-idl.json from that commit, regenerates src/generated, and fails if the committed generated code has drifted. The committed src/generated/*.cpp remains the buildable snapshot the distribution pipeline compiles. The freshly derived catalog is byte-identical in effect: regenerating src/generated from it leaves the committed files unchanged.
…ued bases Extend shape_binary to accept a base-value passed by pointer as the 2nd operand (BASEVAL_PTR_IN, the input mirror of BASEVAL_PTR_RET: PTR_IN minus the temporal handles and collection/box pointers). The operand is marshalled via PTR_IN's BlobTo<Base> and freed after the MEOS call, reusing the existing owned-arg pattern. Making the base value marshallable as a binary operand generates the whole (tcbuffer, cbuffer) surface uniformly from the catalog: the value comparisons (eEq/aEq/tEq and the ?=/%= operators), tDistance/nearestApproachDistance/ nearestApproachInstant, the spatial relationships (eIntersects/eContains/ eCovers/tTouches/...), and the range-point restrictions atValue/minusValue. The set is derived, so Pose/Npoint/Jsonb join automatically once their PTR_IN entry and DuckDB type land. 30 net-new registrations, no hand collisions.
The (Temporal<T>, set-of-T) value restrictions atValues/minusValues were hand-registered for tint/tfloat/ttext. Generate them from the catalog instead: temporal_at_values / temporal_minus_values carry explicit sqlSignatures pairing each temporal type with its element-set type, so a new shape_temporal_restrict_sig registers one overload per pairing whose accessors are both registered Duck types (int/bigint/float/text sets; cbufferset/geomset/etc. are skipped until their set type lands). The pairing is taken straight from the catalog — no name heuristic — and the dispatch runs only for restrictions shape_temporal_span did not claim (atTime), so each function is registered by exactly one path. The generated surface is behaviour-equivalent to the retired hand regs (both call the MEOS set restriction) and adds the previously-missing tbigint overload, now covered by a test.
Extend the sqlSignatures-driven restriction shape to the joint value x time box: tnumber_at_tbox / tnumber_minus_tbox pair each number temporal type with a tbox in their catalog sqlSignatures, so shape_temporal_restrict_sig now accepts a TBox second operand (marshalled via the already-emitted BlobToTbox) and registers one atTbox/minusTbox overload per pairing. The 3-operand stbox restrictions (atStbox, with a border bool) are a separate shape and stay hand for now. Retire the hand atTbox/minusTbox regs (tint/tfloat) and their now-dead Tnumber_at_tbox / Tnumber_minus_tbox wrappers. Behaviour-equivalent — the wrappers called the same MEOS tnumber_at_tbox / tnumber_minus_tbox — and the generated surface adds the previously-missing tbigint overload, now covered by a test.
Bump the MEOS pin to MobilityDB f26ad75164, which names the temporal number span and span set value restrictions atSpan/atSpanset/minusSpan/ minusSpanset. The catalog attaches sqlSignatures to those wrappers, so the generator emits atSpan/atSpanset/minusSpan/minusSpanset for tint, tbigint and tfloat. Point the tint and tfloat span-restriction tests at the canonical names.
Add tools/meos-source-commit.txt as the single source of truth for the MobilityDB commit that both the vcpkg MEOS port and the catalog-derivation workflow use, so the libmeos the binding links and the catalog it generates against always come from the same commit. The vcpkg portfile reads the pin for its REF; generate.yml reads it for the provision-meos ref. Pin to MobilityDB 21eb87481. That commit no longer ships the Arrow C Data Interface, so drop the meos_arrow.h include from meos_wrapper_simple.hpp. The generated UDF surface is unchanged and the full suite passes.
Brings the catalog-driven generator branch up to current main so PR MobilityDB#208 no longer conflicts with its base and CI can run. Main's substantive work (TemporalParquet, geodetic SRID handling, examples) is already present via rebased commits, so the net delta is the ICU-staging Makefile (main's version), the duckdb/extension-ci-tools submodule bumps, and history alignment. The hand tgeompoint predicates main re-added are kept retired -- the generated UDF surface covers them.
…Distance Re-baseline the vendored MEOS source to upstream master aedaa3f06 (source commit, portfile SHA512, port-version), which exports the geo-first point ever/always touches C symbols etouches_geo_tpoint / atouches_geo_tpoint. Select the geo spatial-relationship backing from the catalog sqlSignatures (the source of truth) so eTouches/aTouches on tgeompoint/tgeogpoint bind the point-specific trajectory kernel in both argument orders, instead of the generic per-instant backing that is wrong for moving points. Wire the NxN set-set minDistance(LIST(tgeo), LIST(tgeo)) -> DOUBLE over the four geo temporal types, marshalling each LIST row to a Temporal**. Regenerate src/generated/generated_temporal_udfs.cpp. Full sqllogictest suite green (1399 assertions across 63 test cases).
Add the twelve *Pairs relationships over two arrays of temporal geos: eDwithinPairs/aDwithinPairs, eIntersectsPairs/aIntersectsPairs, eTouchesPairs/aTouchesPairs, eDisjointPairs/aDisjointPairs, and the temporal tDwithinPairs/tIntersectsPairs/tTouchesPairs/tDisjointPairs. Each is generated as a DuckDB scalar UDF over two LIST(<geo>) arguments returning LIST(STRUCT(i INTEGER, j INTEGER[, periods TSTZSPANSET])) that UNNEST to rows. The generator gains a pairs shape on the array-of-temporal input path: it marshals each element BLOB into a Temporal*, calls the MEOS kernel (which returns a flattened [i0,j0,i1,j1,...] int array of index pairs plus, for the temporal variants, a parallel SpanSet* of tstzspansets), and reshapes each row into a DuckDB LIST(STRUCT). Registration is over exactly the geo types the catalog sqlSignatures declare per predicate: eDwithin/eIntersects/eDisjoint over the four geo types, eTouches over tgeometry only, tDwithin over tgeompoint/tgeometry -- the non-uniform MobilityDB surface. Add a functional test covering the value, distance, periods and empty-result shapes.
The <t>Inst / <t>Seq / <t>SeqSet / <t>SeqSetGaps constructor and transform family is a per-type SQL name family, each backed by one base-type-generic MEOS wrapper (temporal_to_tinstant, temporal_to_tsequence, temporal_to_tsequenceset, tsequence_make, tsequenceset_make, tsequenceset_make_gaps). The generator registers each name from the catalog sqlSignatures over the core temporal types (tint, tbigint, tbool, tfloat, ttext) at every arity the signatures declare, and the hand-written registration loop is retired. tsequenceset_make_gaps splits an array of instants into sequences at time gaps (greater than maxt) and value gaps (greater than maxdist), with the per-type argument arities the catalog declares: tbool and ttext take only the time gap, tfloat also takes an interpolation.
The base-type name families tintFromHexWKB/tintFromMFJSON (and the tbigint, tbool, tfloat, ttext siblings) are backed by the base-type-generic MEOS wrappers temporal_from_hexwkb and temporal_from_mfjson. Generate them from the catalog like the constructor family, registering each per-type name over its result temporal type, and retire the hand loop in RegisterWkbFunctions that previously registered only four of them (tbigint was missing). The HexWKB parser copies the DuckDB string into a NUL-terminated std::string before handing it to MEOS, which derives the length with strlen. The MFJSON parser resolves the result temptype from the registered return type and passes it to temporal_from_mfjson.
This was referenced Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The base-type name families
tintFromHexWKB/tintFromMFJSON(and thetbigint,tbool,tfloat,ttextsiblings) are backed by the base-type-generic MEOS wrapperstemporal_from_hexwkbandtemporal_from_mfjson. This generates them from the catalog like the constructor family, registering each per-type name over its result temporal type. It also retires the hand loop inRegisterWkbFunctions, whose four registrations omittbigint; the generated surface covers all five base types.The HexWKB parser copies the DuckDB string into a NUL-terminated
std::stringbefore handing it to MEOS, which derives the length withstrlen. The MFJSON parser resolves the result temptype from the registered return type and passes it totemporal_from_mfjson.Stacked on #211 (reuses its generator dispatch for the temporal I/O families) — the diff resolves to the parser slice once #211 lands. Round-trip tests over all five base types for both formats cover the
tint/tfloat/tbool/ttextsuites.