Skip to content

feat(berlinmod): route the streaming parity matrix through MEOS via JMEOS 1.4 (stacks on #1)#14

Merged
Ar-mie merged 1 commit into
MobilityDB:feat/berlinmod-q1-scaffoldfrom
estebanzimanyi:consolidate/kafka-meos-integration
Jun 19, 2026
Merged

feat(berlinmod): route the streaming parity matrix through MEOS via JMEOS 1.4 (stacks on #1)#14
Ar-mie merged 1 commit into
MobilityDB:feat/berlinmod-q1-scaffoldfrom
estebanzimanyi:consolidate/kafka-meos-integration

Conversation

@estebanzimanyi

@estebanzimanyi estebanzimanyi commented Jun 11, 2026

Copy link
Copy Markdown
Member

Path-B consolidation — PR 2 of 3 (scaffold #1 · MEOS integration · benchmark).

Collapses the former MEOS stack (old #2#10) into one topical commit on top of the reviewed scaffold #1. Routes the BerlinMOD-9 × 3-form processors through MEOS (libmeos via the JMEOS 1.4 bridge) — edwithin_tgeo_geo, eintersects_tgeo_geo, geog_distance, and the set-set join family — and adds the MEOS DSL-tier wirings + per-family facade smoke tests + the canonical BerlinMOD corpus loader.

JMEOS built from source (review fix)

The JMEOS jar and the native libmeos.so are no longer committed to the repository. build-jmeos.sh clones MobilityDB and JMEOS at pinned refs, builds libmeos.so (cmake -DMEOS=ON plus the type families the app uses), builds the jar, installs it into the local Maven repository via mvn install:install-file, and stages libmeos.so for the runtime. The pom resolves JMEOS as an ordinary dependency (no system scope); jar/, lib/ and the build work dir are git-ignored.

Depends on the deliverable PRs, not a pin

The build pins the deliverable PRs' immutable head commit SHAs — MobilityDB #1148 (set-set spatial join, stacked on #1162 clean/geo for the geodetic-disjoint surface) and JMEOS #25 (the org.mobilitydb.meos facade). Both are overridable env vars at the top of the script; switch them to upstream master/main after those PRs are integrated. No ecosystem-pin tag is referenced — a pin is the benchmark/evidence vehicle, not a dependency (and it masks individual-PR gaps).

Irregularities removed vs. the old stack

  • No committed jar/native library; the old stack added ~127k generated lines then deleted them, and bundled jar/JMEOS.jar + lib/libmeos.so.
  • Dropped the dead family-flag Maven profiles (they excluded meos/MeosOps*.java generated files that no longer exist).
  • Removed the invented BerlinMODCorpus.synthetic() corpus — the canonical berlinmod_instants.csv is the only data source.
  • Moved MeosWiringsDemoTopology out of src/main (it used the test-scoped TopologyTestDriver) — same fix as the scaffold's BerlinMODFullMatrixTest.

Verification

./build-jmeos.sh then mvn test11 tests, 0 failures (full matrix + MEOS family smokes + set-set join), against a JMEOS jar built from source at MobilityDB #1148 (fab7025b2, on #1162) + JMEOS #25. MEOS smokes run with reuseForks=false for a clean native lifecycle.

@estebanzimanyi
estebanzimanyi force-pushed the feat/berlinmod-q1-scaffold branch from 346999a to 5e64f44 Compare June 11, 2026 17:55
@estebanzimanyi
estebanzimanyi force-pushed the consolidate/kafka-meos-integration branch from c6f2016 to 6305403 Compare June 11, 2026 17:57
Comment thread kafka-streams-app/jar/JMEOS.jar Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This binary file should not be in the repository, it should be added to the .gitignore The repository should have a Maven file with the proper dependencies needed instead Or have a build-jmeos.shscript instead which clones JMEOS and MobilityDB, compile libmeos.so and generates the jar usingmvn install:install-file`

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed the committed JMEOS.jar and libmeos.so, added them (and the build work dir) to .gitignore, and added build-jmeos.sh at the repo root. It clones MobilityDB and JMEOS at pinned, immutable refs, builds libmeos.so (cmake -DMEOS=ON plus the type families the app uses), builds the jar, installs it into the local Maven repository via mvn install:install-file, and stages libmeos.so for the runtime. The pom now resolves JMEOS as an ordinary dependency (no system scope). Verified end to end: ./build-jmeos.sh then mvn test -> 11/11 green.

Comment thread kafka-streams-app/pom.xml

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are dependencies on the PR #1148 from MobilityDB in the project
This dependency being from an unstable PR can be dangerous if the PR #1148 is ever modified or refused, then this project will have an invalid dependency and not be future proof

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. The build no longer references the open PR #1148 branch. build-jmeos.sh pins MobilityDB to the published, immutable tag ecosystem-pin-2026-06-05e, which already carries the set-set spatial-join surface, so a rebase, force-push or closure of #1148 cannot affect this project. Both pinned refs are overridable env vars at the top of the script, so bumping to a newer line is a one-line change.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction + final state: the build now pins the deliverable PRs' immutable head SHAs, not an ecosystem-pin tag (my earlier reply pinned the pin tag — reverted, since the pin is a benchmark/evidence vehicle, not a dependency). build-jmeos.sh builds libmeos from MobilityDB #1148 (set-set join, now stacked on #1162 clean/geo; head fab7025b2) and the jar from JMEOS #25; both refs are overridable env vars to switch to upstream master/main once integrated. Pinning the real PRs also surfaced a genuine bug the pin had been masking — #1148's set-set aDisjoint was wrong on geodetic data until it was rebased onto #1162; fixed at source. ./build-jmeos.sh then mvn test11/11 green.

@estebanzimanyi
estebanzimanyi force-pushed the consolidate/kafka-meos-integration branch from 1ce3053 to c2ed1dd Compare June 19, 2026 10:29
…MEOS 1.4

Route the BerlinMOD-9 × 3-form processors through MEOS (libmeos via the
JMEOS 1.4 bridge) — edwithin_tgeo_geo, eintersects_tgeo_geo, geog_distance
and the set-set join family — and add the MEOS DSL-tier wirings, the
per-family facade smoke tests and the canonical BerlinMOD corpus loader.

JMEOS and the native libmeos.so are built from source rather than committed
to the repository. build-jmeos.sh clones MobilityDB and JMEOS at the
deliverable PRs' immutable head commit SHAs, builds libmeos.so, builds the
jar and installs it into the local Maven repository, so the pom resolves
JMEOS as an ordinary dependency. It pins MobilityDB #1148 (set-set join,
stacked on #1162 clean/geo) and JMEOS #25 (facade) as overridable env vars;
both are switched to upstream master/main after those PRs are integrated.
@estebanzimanyi
estebanzimanyi force-pushed the consolidate/kafka-meos-integration branch from c2ed1dd to 813643a Compare June 19, 2026 11:39
@Ar-mie
Ar-mie self-requested a review June 19, 2026 11:44
@Ar-mie
Ar-mie merged commit 92d08f9 into MobilityDB:feat/berlinmod-q1-scaffold Jun 19, 2026
Ar-mie added a commit that referenced this pull request Jun 19, 2026
feat(berlinmod): streaming throughput benchmark on the canonical BerlinMOD corpus (stacks on #14)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants