From c2d9c8e4e448380dc7ea01492ea7a860c82188a4 Mon Sep 17 00:00:00 2001 From: deepakpanda93 Date: Mon, 10 Aug 2026 18:00:14 +0530 Subject: [PATCH 1/4] docs(streamer): document checkpoint v2 reset for the Hudi incremental source apache/hudi#12718 changed how --checkpoint is interpreted for the Hudi incremental source once the target table is at table version 8 or higher, but the Checkpointing section still only described the Kafka and DFS formats. A user resetting a checkpoint on such a table hits an IllegalArgumentException with no documentation to consult. Verified end to end on Spark 3.5.7 with Hudi 1.2.0, running HoodieStreamer with HoodieIncrSource against a table-version-9 source table whose four commits have distinct requested and completion times: - No override ingests everything and stores streamer.checkpoint.key.v2 = 20260810122008215, the completion time of the last source commit. So the checkpoint really is completion time based, not request time based. - --checkpoint 20260810122006588, a bare timestamp, fails with "Illegal checkpoint key override ... Valid format is either resumeFromInstantRequestTime: or resumeFromInstantCompletionTime:." - --checkpoint resumeFromInstantRequestTime:20260810122006588 and --checkpoint resumeFromInstantCompletionTime:20260810122007406 both succeed and both ingest exactly ids 31,32,41,42, confirming the two forms name the same boundary and that the request time form is translated to the matching completion time. - Both record the value passed under streamer.checkpoint.reset.key.v2 verbatim. - S3EventsHoodieIncrSource with a bare --checkpoint is not rejected by the parser at all, confirming the DATASOURCES_NOT_SUPPORTED_WITH_CKPT_V2 exclusion in CheckpointUtils; it fails later on unrelated S3 event schema. The upgrade guard is documented from StreamerCheckpointUtils.assertNoCheckpointOverrideDuringUpgradeForHoodieIncSource, which throws when --checkpoint or --ignore-checkpoint is combined with a Hudi incremental source on a run that would upgrade or downgrade the table. Applied to next, 1.2.0, 1.1.1, 1.0.2 and 1.0.1. Not 1.0.0: HoodieIncrSourceCheckpointValUtils does not exist in that release, so the prefixed format would be wrong there. Closes #16813. Co-Authored-By: Claude Opus 5 (1M context) --- website/docs/hoodie_streaming_ingestion.md | 38 +++++++++++++++++++ .../hoodie_streaming_ingestion.md | 38 +++++++++++++++++++ .../hoodie_streaming_ingestion.md | 38 +++++++++++++++++++ .../hoodie_streaming_ingestion.md | 38 +++++++++++++++++++ .../hoodie_streaming_ingestion.md | 38 +++++++++++++++++++ 5 files changed, 190 insertions(+) diff --git a/website/docs/hoodie_streaming_ingestion.md b/website/docs/hoodie_streaming_ingestion.md index 1fb282a5d7185..cf96492912b32 100644 --- a/website/docs/hoodie_streaming_ingestion.md +++ b/website/docs/hoodie_streaming_ingestion.md @@ -320,6 +320,44 @@ If you need to change the checkpoints for reprocessing or replaying data you can - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. +#### Resetting the checkpoint for the Hudi incremental source + +When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by +**completion time** instead of requested instant time, and records it in the commit metadata under +`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which +one it is: + +```shell +# resume after the instant with this requested instant time +--checkpoint resumeFromInstantRequestTime:20250110120000000 + +# resume after the instant with this completion time +--checkpoint resumeFromInstantCompletionTime:20250110120005000 +``` + +Both forms resume from the same position. The request time form is translated internally to the completion time of that +same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim +under `streamer.checkpoint.reset.key.v2`. + +Passing a bare timestamp is rejected: + +```plain +Illegal checkpoint key override `20250110120000000`. Valid format is either +`resumeFromInstantRequestTime:` or `resumeFromInstantCompletionTime:`. +``` + +:::note +`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue +to take a plain `--checkpoint` value regardless of the table version. +::: + +:::caution +Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using +a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the +job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later +run. +::: + ### Transformers Hudi Streamer supports custom transformation on records before writing to storage. This is done by supplying diff --git a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md index 5f559242e8b97..d2779679c98c6 100644 --- a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md @@ -318,6 +318,44 @@ If you need to change the checkpoints for reprocessing or replaying data you can - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. +#### Resetting the checkpoint for the Hudi incremental source + +When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by +**completion time** instead of requested instant time, and records it in the commit metadata under +`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which +one it is: + +```shell +# resume after the instant with this requested instant time +--checkpoint resumeFromInstantRequestTime:20250110120000000 + +# resume after the instant with this completion time +--checkpoint resumeFromInstantCompletionTime:20250110120005000 +``` + +Both forms resume from the same position. The request time form is translated internally to the completion time of that +same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim +under `streamer.checkpoint.reset.key.v2`. + +Passing a bare timestamp is rejected: + +```plain +Illegal checkpoint key override `20250110120000000`. Valid format is either +`resumeFromInstantRequestTime:` or `resumeFromInstantCompletionTime:`. +``` + +:::note +`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue +to take a plain `--checkpoint` value regardless of the table version. +::: + +:::caution +Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using +a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the +job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later +run. +::: + ### Transformers `HoodieStreamer` supports custom transformation on records before writing to storage. This is done by supplying diff --git a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md index c922b4831d079..cf38162673263 100644 --- a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md @@ -318,6 +318,44 @@ If you need to change the checkpoints for reprocessing or replaying data you can - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. +#### Resetting the checkpoint for the Hudi incremental source + +When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by +**completion time** instead of requested instant time, and records it in the commit metadata under +`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which +one it is: + +```shell +# resume after the instant with this requested instant time +--checkpoint resumeFromInstantRequestTime:20250110120000000 + +# resume after the instant with this completion time +--checkpoint resumeFromInstantCompletionTime:20250110120005000 +``` + +Both forms resume from the same position. The request time form is translated internally to the completion time of that +same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim +under `streamer.checkpoint.reset.key.v2`. + +Passing a bare timestamp is rejected: + +```plain +Illegal checkpoint key override `20250110120000000`. Valid format is either +`resumeFromInstantRequestTime:` or `resumeFromInstantCompletionTime:`. +``` + +:::note +`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue +to take a plain `--checkpoint` value regardless of the table version. +::: + +:::caution +Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using +a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the +job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later +run. +::: + ### Transformers `HoodieStreamer` supports custom transformation on records before writing to storage. This is done by supplying diff --git a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md index 286d5765c7518..f57bc396404cd 100644 --- a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md @@ -318,6 +318,44 @@ If you need to change the checkpoints for reprocessing or replaying data you can - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. +#### Resetting the checkpoint for the Hudi incremental source + +When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by +**completion time** instead of requested instant time, and records it in the commit metadata under +`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which +one it is: + +```shell +# resume after the instant with this requested instant time +--checkpoint resumeFromInstantRequestTime:20250110120000000 + +# resume after the instant with this completion time +--checkpoint resumeFromInstantCompletionTime:20250110120005000 +``` + +Both forms resume from the same position. The request time form is translated internally to the completion time of that +same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim +under `streamer.checkpoint.reset.key.v2`. + +Passing a bare timestamp is rejected: + +```plain +Illegal checkpoint key override `20250110120000000`. Valid format is either +`resumeFromInstantRequestTime:` or `resumeFromInstantCompletionTime:`. +``` + +:::note +`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue +to take a plain `--checkpoint` value regardless of the table version. +::: + +:::caution +Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using +a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the +job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later +run. +::: + ### Transformers Hudi Streamer supports custom transformation on records before writing to storage. This is done by supplying diff --git a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md index 7886ff13251b9..d9c74c6451012 100644 --- a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md @@ -320,6 +320,44 @@ If you need to change the checkpoints for reprocessing or replaying data you can - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. +#### Resetting the checkpoint for the Hudi incremental source + +When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by +**completion time** instead of requested instant time, and records it in the commit metadata under +`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which +one it is: + +```shell +# resume after the instant with this requested instant time +--checkpoint resumeFromInstantRequestTime:20250110120000000 + +# resume after the instant with this completion time +--checkpoint resumeFromInstantCompletionTime:20250110120005000 +``` + +Both forms resume from the same position. The request time form is translated internally to the completion time of that +same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim +under `streamer.checkpoint.reset.key.v2`. + +Passing a bare timestamp is rejected: + +```plain +Illegal checkpoint key override `20250110120000000`. Valid format is either +`resumeFromInstantRequestTime:` or `resumeFromInstantCompletionTime:`. +``` + +:::note +`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue +to take a plain `--checkpoint` value regardless of the table version. +::: + +:::caution +Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using +a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the +job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later +run. +::: + ### Transformers Hudi Streamer supports custom transformation on records before writing to storage. This is done by supplying From d92a88e38e9c252d102d3fb53a1512357851041f Mon Sep 17 00:00:00 2001 From: deepakpanda93 Date: Mon, 10 Aug 2026 19:22:33 +0530 Subject: [PATCH 2/4] docs(streamer): name the checkpoint metadata keys correctly, note the 1.0.1 floor Two corrections after running the end-to-end test across every 1.x release. Checkpoint metadata key. The section claimed checkpoints are stored as `streamer.checkpoint.key`, which is not a key Hudi writes. The real ones are deltastreamer.checkpoint.key / deltastreamer.checkpoint.reset_key (StreamerCheckpointV1:29-30, written at :55 and :60) and streamer.checkpoint.key.v2 / streamer.checkpoint.reset.key.v2 (StreamerCheckpointV2:32-33, written at :61 and :66), selected by CheckpointUtils.shouldTargetCheckpointV2 on the write table version. Confirmed by running the streamer both ways against the same source: the default version-9 target commit carries streamer.checkpoint.key.v2, and a target forced to hoodie.write.table.version=6 carries `deltastreamer.checkpoint.key" : "20260810134321000`. StreamerCheckpointV2 also has addV1Props(), which would add the v1 keys alongside the v2 ones, so I checked before wording this as an either/or. It has no callers anywhere in main or test at release-1.2.0, and no commit produced during testing carried both sets, so the mapping is stated plainly without hedging. Version floor. The prefixed --checkpoint form was tested on 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.1.1 and 1.2.0. It works from 1.0.1 onward, on table version 8 (1.0.1, 1.0.2) and table version 9 (1.1.0, 1.1.1, 1.2.0) alike, and both prefixes resolve to the same resume point on the earliest and latest of those. On 1.0.0 it fails outright with "DateTimeParseException: Text 'resumeFromInstant' could not be parsed at index 0", while a bare value is accepted and read as a completion time. Since readers on the current docs may be running 1.0.0, the section now states the 1.0.1 floor rather than leaving them to hit that error. Co-Authored-By: Claude Opus 5 (1M context) --- website/docs/hoodie_streaming_ingestion.md | 6 ++++-- .../version-1.0.1/hoodie_streaming_ingestion.md | 6 ++++-- .../version-1.0.2/hoodie_streaming_ingestion.md | 6 ++++-- .../version-1.1.1/hoodie_streaming_ingestion.md | 6 ++++-- .../version-1.2.0/hoodie_streaming_ingestion.md | 6 ++++-- 5 files changed, 20 insertions(+), 10 deletions(-) diff --git a/website/docs/hoodie_streaming_ingestion.md b/website/docs/hoodie_streaming_ingestion.md index cf96492912b32..0e633554a8ac9 100644 --- a/website/docs/hoodie_streaming_ingestion.md +++ b/website/docs/hoodie_streaming_ingestion.md @@ -312,7 +312,8 @@ Read more in depth about concurrency control in the [concurrency control concept Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`. +Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table +version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: @@ -325,7 +326,8 @@ For Kafka, this is the max # of events to read. When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by **completion time** instead of requested instant time, and records it in the commit metadata under `streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which -one it is: +one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint` +takes a bare completion time. ```shell # resume after the instant with this requested instant time diff --git a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md index d2779679c98c6..a5c5326c4438b 100644 --- a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md @@ -310,7 +310,8 @@ Read more in depth about concurrency control in the [concurrency control concept `HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`. +Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table +version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: @@ -323,7 +324,8 @@ For Kafka, this is the max # of events to read. When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by **completion time** instead of requested instant time, and records it in the commit metadata under `streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which -one it is: +one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint` +takes a bare completion time. ```shell # resume after the instant with this requested instant time diff --git a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md index cf38162673263..08d57c3be25ca 100644 --- a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md @@ -310,7 +310,8 @@ Read more in depth about concurrency control in the [concurrency control concept `HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`. +Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table +version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: @@ -323,7 +324,8 @@ For Kafka, this is the max # of events to read. When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by **completion time** instead of requested instant time, and records it in the commit metadata under `streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which -one it is: +one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint` +takes a bare completion time. ```shell # resume after the instant with this requested instant time diff --git a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md index f57bc396404cd..7881642fc5f54 100644 --- a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md @@ -310,7 +310,8 @@ Read more in depth about concurrency control in the [concurrency control concept Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`. +Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table +version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: @@ -323,7 +324,8 @@ For Kafka, this is the max # of events to read. When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by **completion time** instead of requested instant time, and records it in the commit metadata under `streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which -one it is: +one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint` +takes a bare completion time. ```shell # resume after the instant with this requested instant time diff --git a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md index d9c74c6451012..4d0185757b9d4 100644 --- a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md @@ -312,7 +312,8 @@ Read more in depth about concurrency control in the [concurrency control concept Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`. +Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table +version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: @@ -325,7 +326,8 @@ For Kafka, this is the max # of events to read. When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by **completion time** instead of requested instant time, and records it in the commit metadata under `streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which -one it is: +one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint` +takes a bare completion time. ```shell # resume after the instant with this requested instant time From 05e5b6443aa2d08c55ff1726456cf067f33faa30 Mon Sep 17 00:00:00 2001 From: deepakpanda93 Date: Mon, 10 Aug 2026 20:14:06 +0530 Subject: [PATCH 3/4] docs(streamer): name the checkpoint reset key correctly too Review feedback on apache/hudi#19573: the existing bullet said --checkpoint sets `streamer.checkpoint.reset_key`, which does not reconcile with the `streamer.checkpoint.reset.key.v2` named in the new subsection. It does not reconcile because that key does not exist either. It has the same missing-prefix defect as the checkpoint key line already corrected in d92a88e38e9c: the real reset keys are deltastreamer.checkpoint.reset_key (StreamerCheckpointV1:30, written at :60) and streamer.checkpoint.reset.key.v2 (StreamerCheckpointV2:33, written at :66), chosen by the same write-table-version test. Reworded the bullet to name both, mirroring the phrasing used for the checkpoint key a few lines above. Co-Authored-By: Claude Opus 5 (1M context) --- website/docs/hoodie_streaming_ingestion.md | 2 +- .../versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md | 2 +- .../versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md | 2 +- .../versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md | 2 +- .../versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/website/docs/hoodie_streaming_ingestion.md b/website/docs/hoodie_streaming_ingestion.md index 0e633554a8ac9..02e0127c394ac 100644 --- a/website/docs/hoodie_streaming_ingestion.md +++ b/website/docs/hoodie_streaming_ingestion.md @@ -317,7 +317,7 @@ version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. diff --git a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md index a5c5326c4438b..6b5d9c0f7c41a 100644 --- a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md @@ -315,7 +315,7 @@ version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. diff --git a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md index 08d57c3be25ca..14e4db7d0344b 100644 --- a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md @@ -315,7 +315,7 @@ version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. diff --git a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md index 7881642fc5f54..ec71cafee54a5 100644 --- a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md @@ -315,7 +315,7 @@ version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. diff --git a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md index 4d0185757b9d4..afcd6ee459dc1 100644 --- a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md @@ -317,7 +317,7 @@ version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. From cc3483885c7c6951f913655676a160b5e1adf4f1 Mon Sep 17 00:00:00 2001 From: deepakpanda93 Date: Tue, 11 Aug 2026 10:04:57 +0530 Subject: [PATCH 4/4] docs(streamer): stop tying the checkpoint key to table version alone Review feedback on apache/hudi#19573 challenged the two lines I edited in the Kafka/DFS paragraph, which said the .v2 keys apply at table version 8 and above and the deltastreamer.* keys below that. Testing shows that rule is wrong, though not for the reason given in the review. The reviewer's mechanism does not match release-1.2.0: StreamSync has no createCheckpoint() method, and extractCheckpointMetadata (StreamSync:911-927) uses whatever Checkpoint object the source returned for the next batch, falling back to buildCheckpointFromGeneralSource only when the source supplied none. What actually decides the key is the source's checkpoint type. InputBatch(batch, String) wraps the value in StreamerCheckpointV2 unconditionally (InputBatch:43-44), so sources that hand back a plain string, such as Kafka and DFS, always produce a V2 checkpoint. Confirmed by running ParquetDFSSource twice against the same input: the v2 key is written at table version 9 and equally at table version 6. HoodieIncrSource is different because it returns a Checkpoint object whose type is chosen by table version, which is why the earlier version 6 run of that source produced deltastreamer.checkpoint.key. So the split is by source, and only within the Hudi incremental source by table version. Both lines now name the two keys and say what the choice depends on, without asserting a version rule that does not hold for Kafka or DFS. The version specific detail stays in the incremental source subsection below, where it is backed by tests. Co-Authored-By: Claude Opus 5 (1M context) --- website/docs/hoodie_streaming_ingestion.md | 7 ++++--- .../version-1.0.1/hoodie_streaming_ingestion.md | 7 ++++--- .../version-1.0.2/hoodie_streaming_ingestion.md | 7 ++++--- .../version-1.1.1/hoodie_streaming_ingestion.md | 7 ++++--- .../version-1.2.0/hoodie_streaming_ingestion.md | 7 ++++--- 5 files changed, 20 insertions(+), 15 deletions(-) diff --git a/website/docs/hoodie_streaming_ingestion.md b/website/docs/hoodie_streaming_ingestion.md index 02e0127c394ac..6937ed01d22bc 100644 --- a/website/docs/hoodie_streaming_ingestion.md +++ b/website/docs/hoodie_streaming_ingestion.md @@ -312,12 +312,13 @@ Read more in depth about concurrency control in the [concurrency control concept Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table -version 8 or higher and `deltastreamer.checkpoint.key` below that. +Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under +`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies +depends on the source, and for the Hudi incremental source also on the table version — see below. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. diff --git a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md index 6b5d9c0f7c41a..949e45194cf92 100644 --- a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md @@ -310,12 +310,13 @@ Read more in depth about concurrency control in the [concurrency control concept `HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table -version 8 or higher and `deltastreamer.checkpoint.key` below that. +Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under +`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies +depends on the source, and for the Hudi incremental source also on the table version — see below. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. diff --git a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md index 14e4db7d0344b..edc61eed2a616 100644 --- a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md @@ -310,12 +310,13 @@ Read more in depth about concurrency control in the [concurrency control concept `HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table -version 8 or higher and `deltastreamer.checkpoint.key` below that. +Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under +`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies +depends on the source, and for the Hudi incremental source also on the table version — see below. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. diff --git a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md index ec71cafee54a5..a52dc442bae31 100644 --- a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md @@ -310,12 +310,13 @@ Read more in depth about concurrency control in the [concurrency control concept Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table -version 8 or higher and `deltastreamer.checkpoint.key` below that. +Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under +`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies +depends on the source, and for the Hudi incremental source also on the table version — see below. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. diff --git a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md index afcd6ee459dc1..a4958454598a3 100644 --- a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md +++ b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md @@ -312,12 +312,13 @@ Read more in depth about concurrency control in the [concurrency control concept Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table -version 8 or higher and `deltastreamer.checkpoint.key` below that. +Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under +`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies +depends on the source, and for the Hudi incremental source also on the table version — see below. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: -- `--checkpoint` will set the checkpoint reset key in the commit file to overwrite the current checkpoint — `streamer.checkpoint.reset.key.v2` at table version 8 or higher, `deltastreamer.checkpoint.reset_key` below that. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. +- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read.