Generate the value-domain operator families from the catalog#198
Closed
estebanzimanyi wants to merge 11 commits into
Closed
Generate the value-domain operator families from the catalog#198estebanzimanyi wants to merge 11 commits into
estebanzimanyi wants to merge 11 commits into
Conversation
…aho-mqtt-cpp The Nix environment provides paho-mqtt-cpp as a shared-only library and does not include libmeos. Four changes make the build pass: - flake.nix: add stock paho-mqtt-c + paho-mqtt-cpp to baseThirdPartyDeps; pass -DNES_ENABLE_MEOS=OFF in both defaultPackage and devShell cmakeFlags. - nes-plugins/CMakeLists.txt: introduce NES_ENABLE_MQTT and NES_ENABLE_MEOS options that gate the respective plugin subdirectories via activate_optional_plugin(). - nes-plugins/Sources/MQTTSource, nes-plugins/Sinks/MQTTSink: select PahoMqttCpp::paho-mqttpp3-static when available, fall back to the shared PahoMqttCpp::paho-mqttpp3 target (Nix only ships the shared variant). - nes-physical-operators: gate all MEOS-specific add_plugin calls and the nes-meos link behind if(NES_ENABLE_MEOS) in three CMakeLists files (top-level, Functions/Meos, Aggregation/Function/Meos). - CMakeLists.txt (root): declare option(NES_ENABLE_MEOS) and add_compile_definitions(NES_ENABLE_MEOS) before the nes-* subdirectory loop so the preprocessor symbol is visible to all sibling components. - nes-query-optimizer/LowerToPhysicalWindowedAggregation.cpp: guard the TemporalSequenceAggregationPhysicalFunction include and instantiation with #ifdef NES_ENABLE_MEOS so the translation unit compiles and links when MEOS is disabled.
… build/grammar/QPC glue tools/codegen/ contains codegen_nebula.py, which reads a JSON operator descriptor and emits the four-layer NebulaStream pipeline tuple (logical .hpp/.cpp + physical .hpp/.cpp) per MEOS scalar function, then idempotently injects add_plugin entries into the Meos CMakeLists files, lexer tokens and functionName alternations into AntlrSQL.g4, and dispatch cases into AntlrSQLQueryPlanCreator.cpp — each injection gated by a per-op marker so repeated runs are safe. build_descriptor.py classifies MEOS gap functions by signature into named SHAPEs that select the corresponding physical C++ template; trgeo-descriptor.json is the ready-to-use descriptor for the 34-operator trgeometry family (W148–W149). codegen_aggregations.py handles the separate aggregation four-layer shape. codegen_input.example.json documents the descriptor format. build_local.sh drives the NebulaStream dev-image cmake build without a host C++23 toolchain. The generator reproduces the committed W148–W149 trgeometry surface byte-for-byte from the descriptor. The generated IDL (meos-idl.json) is excluded from the repository and regenerated via MEOS-API run.py against the pinned MEOS headers.
…LIZED*
MEOS-API run.py correctly resolves GSERIALIZED in the IDL; parse_sigs
normalizes `const GSERIALIZED *` to GSERIALIZED* (strip-const + first-token
+ "*"). The four trgeometry shape classifiers that match geometry arguments
use GSERIALIZED* throughout:
- trgeometry_geo_predicate: (Temporal*, int*) → (Temporal*, GSERIALIZED*)
- geo_trgeometry_predicate: (int*, Temporal*) → (GSERIALIZED*, Temporal*)
- trgeometry_geo_dwithin: (Temporal*, int*, double) → (Temporal*, GSERIALIZED*, double)
- trgeometry_nad geo-branch: (Temporal*, int*) → (Temporal*, GSERIALIZED*)
The trgeometry_nad two-temporal branch adds fn.endswith("_trgeometry") so
nad_trgeometry_tpoint (Temporal*, Temporal*) is excluded, matching the
committed descriptor's scope.
Equivalence probe: build_descriptor.py against the master IDL (4492 fns)
reproduces the committed trgeo-descriptor.json 34-operator set byte-for-byte.
Genuine int* out-params (trgeometry_instants/segments/sequences/stboxes, all
with non-int return types) are unaffected. nad_trgeometry_tpoint is correctly
excluded.
…lways verbs
cmp_scalar_tempfirst and cmp_scalar_scalarfirst matched on signature alone, so a nearest-approach
function like nad_tint_int (int(Temporal*, int)) was misclassified as a comparison operator. Require
the ever/always + {eq,ne,lt,le,gt,ge} naming so only comparison functions match.
The physical .cpp emitter carried a hardcoded two-temporal serialize-on-return template (f(Temporal*, Temporal*) -> Temporal* over hex-WKB operands) separate from the value-return extras dispatch, so every new extra-arg kind (scalar, static geometry, box, second temporal) needed its own parallel wkb template. Extract the shared operand marshalling into _generic_call_extras() and route both the value-return path (assemble_generic_physical) and the serialize-on-return path (assemble_wkb_output) through it. assemble_wkb_output is now general over the primary temporal plus any extra_args, honouring scalar_first/box_first ordering, and the per-kind wkb templates collapse into one GENERIC_PHYSICAL_WKB_GENERAL_TEMPLATE. Add a temporal_transform_wkb catch-all classifier: any Temporal*-returning per-event transform whose operands are all marshallable (scalar/geom/box/temporal) is emitted via this path; unsupported operands (out-params, sequence/instant operands, affine matrices, arity>3 stateful, bare spans) return None so they stay measurable structural residue rather than silent drops.
build_descriptor.py re-parsed a raw MEOS header dump (--sigs) plus a hand-maintained gap list (--gap) to discover function signatures — bypassing the MEOS-API catalog and breaking the codegen chain MEOS -> MEOS-API -> binding (a binding must be a projection of meos-idl.json, never of the headers directly). Add load_catalog()/_norm_ctype() and a --catalog input that reads meos-idl.json as the single signature + surface authority. With no --gap, every api=public function is a candidate and the SHAPE classifiers decide which are Nebula-streamable; the catalog's network.exposable flag is NOT used for that (it is a REST-server decoder constraint, e.g. reason "no-decoder:int64_t", and int64 columns stream fine in Nebula). --gap/--wired stay as optional subset/exclusion filters. The catalog is derived (run.py / provision-meos), never committed (.gitignore). Reconcile the scalar-arg classifiers to the catalog-canonical token int64_t (was the header spelling int64), and add the geo-first tgeo predicate shape + geom_first call ordering so acovers_geo_tgeo(geo, temp) and edwithin_geo_tgeo(geo, temp, dist) emit with the geometry leading. Running in --catalog mode over master 205449f90f: 4023 public candidates -> 763 generatable by the current shapes.
Comparisons and restrictions whose operand is a MEOS base type (Cbuffer / Npoint / Nsegment / Pose / Jsonb) were residue: the classifiers only knew numeric scalars, static geometries, and boxes. But a base-type operand carries and parses exactly like a box literal — VARSIZED text -> <type>_in -> freed — so no new emission machinery is needed. Add _BASETYPE_PARSER (base cType -> cpp type / text parser / header), a basetype_cmp classifier for int fn(<Base>*, Temporal*) / (Temporal*, <Base>*) emitting a box-kind extra (geom_first when the base value leads the MEOS call), and teach temporal_transform_wkb to map a base-type operand to the same box-kind extra so Temporal*-returning restrictions (tcbuffer_restrict_cbuffer, tdistance_tnpoint_npoint, ...) generate too. Catalog mode at master 205449f90f: 763 -> 825 generatable (+62 = 32 base-type comparisons + ~30 base-type restriction transforms), verified to emit the correct <base>_in parse and MEOS call argument order.
Transforms returning a GSERIALIZED* geometry (tpoint_trajectory, *_convex_hull, *_traversed_area, shortestline_*, twcentroid, tgeo_start/end_value) were residue: the serialize-on-return path only knew how to emit a Temporal* result. Generalize assemble_wkb_output with an output_kind: the Temporal* case serializes via temporal_as_hexwkb (unchanged); the new "geom" case serializes the GSERIALIZED* result via geo_as_hexewkb(res, NULL) — hex-EWKB, NDR, SRID-preserving — which round-trips through the geometry input path (StaticGeometry -> geom_in, which accepts hex-EWKB). Teach temporal_transform_wkb to accept a GSERIALIZED* return and tag output_kind="geom"; all the existing operand kinds (temporal / static geometry / base type / scalar) carry over, so mixed forms like shortestline_tnpoint_npoint(temp, npoint) and shortestline_tgeo_geo(temp, geo) generate too. Catalog mode at master 205449f90f: 825 -> 857 generatable (+32). Verified the emitted geom-output physical .cpp (GSERIALIZED* res -> geo_as_hexewkb -> arena VARSIZED, free(res), meos_geo.h included) and that the Temporal* output path is byte-unchanged.
… kind Accessors returning a MEOS base value (tcbuffer_start_value -> Cbuffer, tnpoint_end_value -> Npoint, tpose_start_value -> Pose, tjsonb_start_value -> Jsonb) were residue: the serialize-on-return path emitted only Temporal* and GSERIALIZED* results. Add a third output_kind, "basetype": serialize the typed base result to text via the public <base>_out(res, 15) wrapper (15 = OUT_DEFAULT_DECIMAL_DIGITS, MEOS's own default; jsonb_out takes no precision arg), which round-trips through <base>_in on the consuming input. Add _BASETYPE_OUT (return type -> serialize call + header) and a basetype_accessor classifier for <Base>* fn(Temporal*). Datum-returning accessors (temporal_start_value) are internal typed-wrapper territory and never match (ret "Datum" is not a base return type). Catalog mode at master 205449f90f: 857 -> 865 generatable (+8), verified the emitted <base>_out serialize and that the internal Datum accessors stay excluded.
The descriptor built each operator's #include from a hand-maintained symbol->header map (_transform_extra_headers plus two inline guesses). The map mis-routed the trgeometry family to meos_pose.h (its symbols live in meos_rgeo.h) and had no entry for the json, h3, or quadbin families, so those operators referenced undeclared identifiers and failed to compile against a full -DALL libmeos. Derive the declaring header for each meos_call from the single-source catalog (meos-idl.json 'file' field) instead, and only emit an operator whose symbol is declared in an installed public umbrella header (meos.h / meos_<family>.h, excluding meos_internal*). Symbols the catalog attributes to an internal source header (validators such as ensure_valid_*, internal workers such as tinterrel_tgeo_geo) are reason-marked residue rather than emitted with an unbuildable include; the honest fix for a genuinely useful such symbol is to export it in a MEOS umbrella upstream, which then flows to an operator automatically. Generated surface 865 -> 805 (60 internal-header symbols dropped). All 805 meos_calls verified declared in their chosen umbrella at the catalog commit; the 66 previously-failing json/h3/quadbin/trgeometry operator TUs compile and link clean against the -DALL libmeos.
Add generator shapes for the value-domain span/set/spanset/box surface, carried as VARSIZED text fields and streamed per event: predicates, typed accessors, complex and serialize-on-return accessors, set operations, typed transforms, and date/tstz predicates and accessors. Read the concrete SQL return/argument tokens and every public *_in parser and *_out serializer arity straight from the catalog, so a shape resolves result element types and serializer signatures without name-parsing. Skip pointer-to-pointer (array / out-parameter) signatures, which have no per-event streaming form.
Member
Author
|
The value-domain generator shapes live on the single generator-completion branch. The clean topical PRs come from the complete generated surface, not standalone slices. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds generator shapes for the value-domain span/set/spanset/box surface,
carried as VARSIZED text fields and streamed per event:
date/tstz spans, sets and span-sets, both operand orders)
*_outresults)Reads the concrete SQL return/argument tokens and every public
*_inparser and
*_outserializer arity straight from the catalog, so a shaperesolves result element types and serializer signatures without
name-parsing. Skips pointer-to-pointer (array / out-parameter) signatures,
which have no per-event streaming form.
Builds on the catalog-driven generator infrastructure: catalog
consumption, a single serialize-on-return assembler, the base-type and
geometry output kinds, and catalog-gated operator headers. Those
infrastructure commits appear in the diff, since they are not present on
the base branch.