Skip to content

feat(berlinmod): streaming benchmark + regenerate the MeosOps facades from master-tracked JMEOS#31

Merged
Ar-mie merged 3 commits into
MobilityDB:mainfrom
estebanzimanyi:consolidate/flink-benchmark
Jul 9, 2026
Merged

feat(berlinmod): streaming benchmark + regenerate the MeosOps facades from master-tracked JMEOS#31
Ar-mie merged 3 commits into
MobilityDB:mainfrom
estebanzimanyi:consolidate/flink-benchmark

Conversation

@estebanzimanyi

@estebanzimanyi estebanzimanyi commented Jun 11, 2026

Copy link
Copy Markdown
Member

Adds the BerlinMOD streaming throughput benchmark and regenerates the in-repo org.mobilitydb.meos.MeosOps* facades, with the vendored JMEOS.jar + libmeos.so rebuilt from the current MobilityDB master surface via the canonical chain MEOS → MEOS-API catalog → JMEOS (track-master-surface).

  • tools/codegen_facades.py emits the 57 MeosOps* forwarder facades from the bundled JMEOS functions.GeneratedFunctions ∩ the streaming-relevance baseline.
  • MeosSetSetJoin recovered as source.
  • Bundled JMEOS.jar/libmeos.so built from MobilityDB master (4493-function catalog).

Verification

mvn test10 tests, 0 failures, 0 errors, 0 skipped.

This was referenced Jun 11, 2026
@estebanzimanyi
estebanzimanyi force-pushed the consolidate/flink-meos-integration branch from 6d06a5f to db2e51b Compare June 11, 2026 18:04
@estebanzimanyi
estebanzimanyi force-pushed the consolidate/flink-benchmark branch 3 times, most recently from 5051555 to 7de1706 Compare June 14, 2026 21:10
@estebanzimanyi
estebanzimanyi force-pushed the consolidate/flink-benchmark branch from 03f2755 to 5697d6d Compare June 22, 2026 23:57
estebanzimanyi added a commit to estebanzimanyi/MobilityFlink that referenced this pull request Jun 25, 2026
Add tools/pin/compose-order.txt (the canonical fold manifest; MobilityDB#30 merged, MobilityDB#31
carries the facade generator + benchmark) and GENERATION.md (the per-binding
generator policy: MobilityFlink binds the JMEOS jar and generates the
org.mobilitydb.meos facades via tools/codegen_facades.py).
@JashanReel

Copy link
Copy Markdown
Collaborator

This PR is trying to merge into MobilityDB:consolidate/flink-meos-integration from estebanzimanyi:consolidate/flink-benchmark, but shouldn't it be into MobilityDB:main since this PR (#31) stacks on #30 which was only merged into MobilityDB:main ?

…inMOD corpus

Adds the BerlinMOD streaming benchmark harness (BerlinMODBenchmark, per
Q-form throughput over the DataStream topology), the benchmark-results
document, and the Kafka producer script for feeding the corpus.

The benchmark runs on the canonical BerlinMOD corpus only (--csv
berlinmod_instants.csv); the invented synthetic fallback is gone.
… jar + libmeos from MobilityDB master

Generate the org.mobilitydb.meos.MeosOps* streaming facades from the
bundled JMEOS jar via tools/codegen_facades.py, recover MeosSetSetJoin
as source, and vendor the JMEOS.jar + libmeos.so built from the current
MobilityDB master surface (MEOS -> MEOS-API catalog -> JMEOS).
@estebanzimanyi
estebanzimanyi force-pushed the consolidate/flink-benchmark branch from 5697d6d to 542288b Compare July 8, 2026 09:41
@estebanzimanyi
estebanzimanyi changed the base branch from consolidate/flink-meos-integration to main July 8, 2026 09:42
@estebanzimanyi estebanzimanyi changed the title feat(berlinmod): streaming throughput benchmark on the canonical BerlinMOD corpus (stacks on #30) feat(berlinmod): streaming benchmark + regenerate the MeosOps facades from master-tracked JMEOS Jul 8, 2026
@Ar-mie

Ar-mie commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

The documentation is not updated to fir the code
In the main() the env.execute(name) is not in a try catch
Maybe for the t.keyBy(x -> 0) in the Q5-continuous it should be specified that all events are in one subtask on purpose
The tests mentioned in the PR description mvn test → 10 tests, 0 failures seem not to include the BerlinMODBenchmark

- Align the BerlinMODBenchmark Javadoc to the code: the corpus is the real
  BerlinMOD instants CSV (--csv, required); drop the stale --vehicles/--events
  synthetic-mode reference and the dead 'synthetic' corpus label.
- Wrap each cell's Flink job (env.execute) in try-catch so one failing cell is
  recorded and the matrix continues instead of aborting.
- Comment the deliberate keyBy(x -> 0) single-subtask routing in Q5/Q9 (the
  pairwise-meet / all-pairs queries need a global, non-parallel view).
- Cover the benchmark in the suite: add BerlinMODBenchmarkTest (a keyed cell and
  a keyBy(x -> 0) cell over a small in-memory corpus), and remove the nine
  misplaced BerlinMODQ*LocalTest main-drivers that sat in src/main (test-named,
  never run by mvn test, and dependent on an uncommitted sample resource).
@estebanzimanyi

Copy link
Copy Markdown
Member Author

Thanks for the review — all four addressed in 0303853:

  1. Docs — the BerlinMODBenchmark Javadoc referenced a --vehicles/--events synthetic mode the code no longer has; realigned it to the actual --csv (required) corpus and removed the dead synthetic label.
  2. env.execute not in try-catch — each cell's job is now wrapped in try-catch, so a failing cell is recorded (ERROR row) and the matrix continues instead of aborting.
  3. keyBy(x -> 0) in Q5-continuous — added a comment: Q5 (pairwise meet) and Q9 (all-pairs) route the whole stream to a single subtask on purpose (a global, non-parallel view).
  4. Benchmark not in the tests — added BerlinMODBenchmarkTest (runs a keyed cell and a keyBy(x -> 0) cell through runCell on a small in-memory corpus), so the benchmark path is now exercised by mvn test (12 tests, 0 failures). Also removed the nine BerlinMODQ*LocalTest drivers that sat under src/main — test-named but never run by surefire, and dependent on an uncommitted sample resource.

@Ar-mie
Ar-mie merged commit c40c4b2 into MobilityDB:main Jul 9, 2026
Ar-mie added a commit that referenced this pull request Jul 9, 2026
build(flink): migrate to Apache Flink 2.0 (stacks on #31)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants