build(flink): migrate to Apache Flink 2.0 (stacks on #31)#33
Merged
Conversation
…inMOD corpus Adds the BerlinMOD streaming benchmark harness (BerlinMODBenchmark, per Q-form throughput over the DataStream topology), the benchmark-results document, and the Kafka producer script for feeding the corpus. The benchmark runs on the canonical BerlinMOD corpus only (--csv berlinmod_instants.csv); the invented synthetic fallback is gone.
… jar + libmeos from MobilityDB master Generate the org.mobilitydb.meos.MeosOps* streaming facades from the bundled JMEOS jar via tools/codegen_facades.py, recover MeosSetSetJoin as source, and vendor the JMEOS.jar + libmeos.so built from the current MobilityDB master surface (MEOS -> MEOS-API catalog -> JMEOS).
- Align the BerlinMODBenchmark Javadoc to the code: the corpus is the real BerlinMOD instants CSV (--csv, required); drop the stale --vehicles/--events synthetic-mode reference and the dead 'synthetic' corpus label. - Wrap each cell's Flink job (env.execute) in try-catch so one failing cell is recorded and the matrix continues instead of aborting. - Comment the deliberate keyBy(x -> 0) single-subtask routing in Q5/Q9 (the pairwise-meet / all-pairs queries need a global, non-parallel view). - Cover the benchmark in the suite: add BerlinMODBenchmarkTest (a keyed cell and a keyBy(x -> 0) cell over a small in-memory corpus), and remove the nine misplaced BerlinMODQ*LocalTest main-drivers that sat in src/main (test-named, never run by mvn test, and dependent on an uncommitted sample resource).
Bump flink.version to 2.0.0 and migrate the APIs removed in Flink 2.0: - windowing.time.Time -> java.time.Duration - StreamExecutionEnvironment.fromCollection -> fromData - RichFunction.open(Configuration) -> open(OpenContext) - RichSinkFunction/addSink -> Sink v2 (createWriter) / executeAndCollect - decouple the externalized flink-connector-kafka (4.0.1-2.0) - drop the unused flink-java and flink-connector-jdbc dependencies - delete the dead AISTestSource (legacy SourceFunction) Gate the Sedona comparison queries behind a -Psedona profile because Apache Sedona has no Flink-2.0 build yet (latest targets Flink 1.19); the MEOS/BerlinMOD path builds on Flink 2.0 by default.
estebanzimanyi
force-pushed
the
feat/flink-2-migration
branch
from
July 8, 2026 14:39
30213b8 to
3a2897d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on #31 — the first two commits are #31 (facade regen + master-tracked JMEOS jar); this PR adds the Flink 2.0 migration on top. When #31 merges, this reduces to the single
build(flink)commit.What
Migrates the processor from Flink 1.16 to Apache Flink 2.0, resolving the version skew with the Java 21 Docker base (Flink 1.16 supported only Java 8/11; Flink 2.0 targets Java 17/21, matching the
java-21-amazon-correttoimage).Removed-API migrations:
windowing.time.Time→java.time.DurationStreamExecutionEnvironment.fromCollection→fromDataRichFunction.open(Configuration)→open(OpenContext)RichSinkFunction/addSink→ Sink v2 (createWriter) /executeAndCollectflink-connector-kafka(4.0.1-2.0)flink-javaandflink-connector-jdbcdependenciesAISTestSource(legacySourceFunction)Sedona
Apache Sedona has no Flink-2.0 build yet (latest 1.9.0 still targets Flink 1.19), so the
sedona/**comparison queries are gated behind a-PsedonaMaven profile and excluded from the default build. Re-enable with-Psedonaonce an upstream Sedona/Flink-2.0 release exists.Verification
mvn clean test→ 10 tests, 0 failures, 0 errors, 0 skipped against the master-tracked JMEOS jar + libmeos.