Add: Stream-Stream Joins in Real-Time Mode — Ad Click Attribution demo - #99
Conversation
HeartSaVioR
left a comment
HeartSaVioR
left a comment
There was a problem hiding this comment.
Looked through RTM-SSJ.py as requested; looks OK to me, assuming @muralitalluri-db ran this manually to confirm the code. Let's just use a (couple) seconds or just 0 second for processing time trigger.
|
|
||
| Joining two live streams is one of the most common — and most latency-sensitive — patterns in operational streaming. Until now, doing it with sub-second latency in Spark meant reaching for a second engine. RTM closes that gap: the **same Structured Streaming stream-stream join you already write** now runs in Real-Time Mode with a **single trigger change**. | ||
|
|
||
| You bring **Kafka**, **Unity Catalog**, and a **Databricks Runtime** that supports this workload; we provide the **notebooks** and the **data generator / replay** path so a team can reproduce it in their own workspace. **RTM stream-stream join requires latest DBR 18.x** (18.2+ recommended). |
There was a problem hiding this comment.
I'd just recommend DBR 18 (18.3) rather than pointing to non-LTS.
| - **Inner join only** (outer joins are not supported in RTM). | ||
| - **`update` output mode only.** | ||
| - **Both sides require watermarks**, and the join must include an **explicit time bound** so state stays bounded. | ||
| - A few **Spark configurations** enable it (shown below), plus the standard cluster-level RTM requirements (classic compute, no autoscaling, no Photon, DBR 18+). |
There was a problem hiding this comment.
This will be changed sooner with SDP serverless (maybe it could work today), but I'm OK to follow the official guide doc.
|
|
||
| ### 1. Databricks workspace / compute | ||
|
|
||
| RTM stream-stream join requires a **classic** cluster on **latest DBR 18.x** (18.2+ recommended so `update` output mode is supported for stream-stream joins in **both** modes) with: |
There was a problem hiding this comment.
same, why not just point to DBR 18 LTS?
| └── Write_RTM_attributed_clicks_to_delta.py # Optional: attributed_clicks (Kafka) -> Delta for SQL latency | ||
| ``` | ||
|
|
||
| **Fast path:** |
There was a problem hiding this comment.
I'm not sure I understand what this means; quickstart?
|
|
||
| | Setting | RTM | MBM | | ||
| |---------|-----|-----| | ||
| | Trigger | `trigger(realTime="5 minutes")` | `trigger(processingTime="0.5 seconds")` | |
There was a problem hiding this comment.
I'd give a couple seconds to MBM; if the batch duration can be stably 500ms in MBM we shouldn't have started the project of RTM.
There was a problem hiding this comment.
If you meant to benchmark without waiting to trigger the next batch, maybe 0 seconds to be explicit.
| | p99 | 2,750 ms | 167 ms | ~16× | | ||
| | max | 6,084 ms | 2,599 ms | ~2.3× | | ||
|
|
||
| **MBM's ~1-second batches are its latency floor** — a matched record can't be emitted until its batch completes. **RTM flows records through continuously** (single-digit-ms min), so median drops ~24× and p99 ~16×, with **zero records dropped** and **bounded state**. Your cluster will differ — treat this as a baseline and reproduce with the listener / `debug.sql`. |
There was a problem hiding this comment.
The explanation of MBM isn't actually accurate - it's only applied to stage barrier and the last stage to sink won't have such a latency floor for writing to Kafka. Though it might be acceptable if this is to simplify the explanation.
3 participants
Adds a self-serve sample demonstrating stream-stream join support in Apache Spark Real-Time Mode (DBR 18+), using a real-time ad click attribution use case (impressions joined to clicks, inner time-bounded join). Includes a data generator + Delta replay harness (~30K events/sec), Kafka ingest notebooks, the RTM/MBM join notebook (only the trigger changes), optional Delta landing + debug.sql for latency percentiles, and a README. Workspace-specific values are left as placeholders. Companion blog URL is a TODO to add on publish.