Generate temporal-to-temporal transform operators via hex-WKB serialization#194
Open
estebanzimanyi wants to merge 5 commits into
Open
Generate temporal-to-temporal transform operators via hex-WKB serialization#194estebanzimanyi wants to merge 5 commits into
estebanzimanyi wants to merge 5 commits into
Conversation
…aho-mqtt-cpp The Nix environment provides paho-mqtt-cpp as a shared-only library and does not include libmeos. Four changes make the build pass: - flake.nix: add stock paho-mqtt-c + paho-mqtt-cpp to baseThirdPartyDeps; pass -DNES_ENABLE_MEOS=OFF in both defaultPackage and devShell cmakeFlags. - nes-plugins/CMakeLists.txt: introduce NES_ENABLE_MQTT and NES_ENABLE_MEOS options that gate the respective plugin subdirectories via activate_optional_plugin(). - nes-plugins/Sources/MQTTSource, nes-plugins/Sinks/MQTTSink: select PahoMqttCpp::paho-mqttpp3-static when available, fall back to the shared PahoMqttCpp::paho-mqttpp3 target (Nix only ships the shared variant). - nes-physical-operators: gate all MEOS-specific add_plugin calls and the nes-meos link behind if(NES_ENABLE_MEOS) in three CMakeLists files (top-level, Functions/Meos, Aggregation/Function/Meos). - CMakeLists.txt (root): declare option(NES_ENABLE_MEOS) and add_compile_definitions(NES_ENABLE_MEOS) before the nes-* subdirectory loop so the preprocessor symbol is visible to all sibling components. - nes-query-optimizer/LowerToPhysicalWindowedAggregation.cpp: guard the TemporalSequenceAggregationPhysicalFunction include and instantiation with #ifdef NES_ENABLE_MEOS so the translation unit compiles and links when MEOS is disabled.
… build/grammar/QPC glue tools/codegen/ contains codegen_nebula.py, which reads a JSON operator descriptor and emits the four-layer NebulaStream pipeline tuple (logical .hpp/.cpp + physical .hpp/.cpp) per MEOS scalar function, then idempotently injects add_plugin entries into the Meos CMakeLists files, lexer tokens and functionName alternations into AntlrSQL.g4, and dispatch cases into AntlrSQLQueryPlanCreator.cpp — each injection gated by a per-op marker so repeated runs are safe. build_descriptor.py classifies MEOS gap functions by signature into named SHAPEs that select the corresponding physical C++ template; trgeo-descriptor.json is the ready-to-use descriptor for the 34-operator trgeometry family (W148–W149). codegen_aggregations.py handles the separate aggregation four-layer shape. codegen_input.example.json documents the descriptor format. build_local.sh drives the NebulaStream dev-image cmake build without a host C++23 toolchain. The generator reproduces the committed W148–W149 trgeometry surface byte-for-byte from the descriptor. The generated IDL (meos-idl.json) is excluded from the repository and regenerated via MEOS-API run.py against the pinned MEOS headers.
…LIZED*
MEOS-API run.py correctly resolves GSERIALIZED in the IDL; parse_sigs
normalizes `const GSERIALIZED *` to GSERIALIZED* (strip-const + first-token
+ "*"). The four trgeometry shape classifiers that match geometry arguments
use GSERIALIZED* throughout:
- trgeometry_geo_predicate: (Temporal*, int*) → (Temporal*, GSERIALIZED*)
- geo_trgeometry_predicate: (int*, Temporal*) → (GSERIALIZED*, Temporal*)
- trgeometry_geo_dwithin: (Temporal*, int*, double) → (Temporal*, GSERIALIZED*, double)
- trgeometry_nad geo-branch: (Temporal*, int*) → (Temporal*, GSERIALIZED*)
The trgeometry_nad two-temporal branch adds fn.endswith("_trgeometry") so
nad_trgeometry_tpoint (Temporal*, Temporal*) is excluded, matching the
committed descriptor's scope.
Equivalence probe: build_descriptor.py against the master IDL (4492 fns)
reproduces the committed trgeo-descriptor.json 34-operator set byte-for-byte.
Genuine int* out-params (trgeometry_instants/segments/sequences/stboxes, all
with non-int return types) are unaffected. nad_trgeometry_tpoint is correctly
excluded.
…lways verbs
cmp_scalar_tempfirst and cmp_scalar_scalarfirst matched on signature alone, so a nearest-approach
function like nad_tint_int (int(Temporal*, int)) was misclassified as a comparison operator. Require
the ever/always + {eq,ne,lt,le,gt,ge} naming so only comparison functions match.
…zation Generate the 15 unary per-event transforms whose result is itself a temporal value that cannot be flattened to a scalar column: the number-domain tfloat_cos/sin/tan trigonometric maps and the tfloat_to_tbigint / tint_to_tbigint type conversions, together with tgeompoint_to_tnpoint, trgeometry_to_tgeometry, trgeometry_centroid/speed/cumulative_length, and the tpose speed/pitch/roll/yaw/angular_speed accessors, across the logical and physical layers, with matching grammar tokens, parser dispatch, and CMake entries. Such a transform takes one hex-WKB temporal operand and returns a temporal, so it follows the same serialize-on-return round-trip the two-operand cross-vehicle form already uses: parse the operand with temporal_from_hexwkb, call the MEOS function, and serialize the temporal result with temporal_as_hexwkb into an arena-allocated variable-sized field. A single-operand physical template and its classifier reuse that round-trip; scalar-carrying transforms resolve to the value-extracting path first, so this covers only the non-scalar temporal results.
estebanzimanyi
force-pushed
the
feat/nebula-temporal-transform-wkb
branch
from
July 11, 2026 10:52
66bd389 to
3a51091
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generate the 15 unary per-event transforms whose result is itself a temporal value that cannot be flattened to a scalar column, across the logical and physical layers with matching grammar tokens, parser dispatch, and CMake entries:
tfloat_cos,tfloat_sin,tfloat_tan(trigonometric maps) andtfloat_to_tbigint,tint_to_tbigint(type conversions)tgeompoint_to_tnpoint,trgeometry_to_tgeometry,trgeometry_centroid,trgeometry_speed,trgeometry_cumulative_lengthtpose_speed,tpose_pitch,tpose_roll,tpose_yaw,tpose_angular_speedSuch a transform takes one hex-WKB temporal operand and returns a temporal, so it follows the same serialize-on-return round-trip the two-operand cross-vehicle form uses: parse the operand with
temporal_from_hexwkb, call the MEOS function, and serialize the temporal result withtemporal_as_hexwkbinto an arena-allocated variable-sized field. A single-operand physical template and its classifier reuse that round-trip; scalar-carrying transforms resolve to the value-extracting path first, so this covers only the non-scalar temporal results.The classifier (
temporal_unary_transform_wkbshape inbuild_descriptor.py) and the physical template live in the IDL-driven generator; the operators are regenerated, not hand-written. Each physical operator builds against a-DMEOS=ON -DALL=ONlibmeos.