Skip to content

docs(streamer): document checkpoint v2 reset for the Hudi incremental source #19573

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
deepakpanda93 wants to merge 4 commits into
base: asf-site
Choose a base branch
from
+215 −10
Open

docs(streamer): document checkpoint v2 reset for the Hudi incremental source #19573

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c2d9c8e
docs(streamer): document checkpoint v2 reset for the Hudi incremental…
deepakpanda93 Aug 10, 2026
d92a88e
docs(streamer): name the checkpoint metadata keys correctly, note the…
deepakpanda93 Aug 10, 2026
05e5b64
docs(streamer): name the checkpoint reset key correctly too
deepakpanda93 Aug 10, 2026
cc34838
docs(streamer): stop tying the checkpoint key to table version alone
deepakpanda93 Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 43 additions & 2 deletions website/docs/hoodie_streaming_ingestion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,14 +312,55 @@ Read more in depth about concurrency control in the [concurrency control concept
Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under
`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies
depends on the source, and for the Hudi incremental source also on the table version — see below.

If you need to change the checkpoints for reprocessing or replaying data you can use the following options:

- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.

@hudi-agent hudi-agent Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 It might help to reconcile the key names here. The existing bullet says --checkpoint sets streamer.checkpoint.reset_key, while the new subsection says the value is recorded under streamer.checkpoint.reset.key.v2 for table version 8+. A reader inspecting commit metadata may be unsure which key applies to their table. Consider noting that reset.key.v2 is used for table version 8+ and the older reset_key for versions below that, mirroring the streamer.checkpoint.key.v2 vs deltastreamer.checkpoint.key distinction you already added just above.

⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.

@deepakpanda93 deepakpanda93 Aug 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right that these do not reconcile — and the reason is that streamer.checkpoint.reset_key is not a real key either. Fixed in 05e5b64.

It has the same missing-prefix defect as the checkpoint key line I corrected in d92a88e. The actual reset keys, at release-1.2.0:

Checkpoint Reset key Defined / written
request time deltastreamer.checkpoint.reset_key StreamerCheckpointV1.java:30, written at :60
completion time streamer.checkpoint.reset.key.v2 StreamerCheckpointV2.java:33, written at :66

Same CheckpointUtils.shouldTargetCheckpointV2 test on the write table version that selects between deltastreamer.checkpoint.key and streamer.checkpoint.key.v2. So the bullet was not describing an older key superseded by the new one — it was naming a key Hudi never writes, which is exactly why a reader inspecting commit metadata would not find it.

The bullet now reads:

--checkpoint will set the checkpoint reset key in the commit file to overwrite the current checkpoint — streamer.checkpoint.reset.key.v2 at table version 8 or higher, deltastreamer.checkpoint.reset_key below that.

which mirrors the phrasing used for the checkpoint key a few lines above, so the two lines now agree with each other and with the new subsection. Empirically confirmed earlier in this PR: the version-9 target commit carries streamer.checkpoint.reset.key.v2 with the prefixed override verbatim, and a target forced to hoodie.write.table.version=6 carries the deltastreamer.* pair.

Build is clean with the warning set byte-identical to the baseline.

@hudi-agent hudi-agent Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Same root cause as the intro line: this bullet now says --checkpoint sets streamer.checkpoint.reset.key.v2 at table version 8+, but buildCheckpointFromConfigOverride() only returns the V2-capable (UnresolvedStreamerCheckpointBasedOnCfg) checkpoint for the HoodieIncrSource family — for Kafka/DFS sources it returns StreamerCheckpointV1, whose getCheckpointCommitMetadata() writes the reset value under deltastreamer.checkpoint.reset_key regardless of table version. Since this bullet is describing the Kafka-format --checkpoint value, the reset key it will actually see is deltastreamer.checkpoint.reset_key, not the .v2 one. Consider limiting the .v2 reset-key mention to the Hudi incremental source (where the new subsection already covers it) and keeping the V1 key for the Kafka/DFS bullet. This is a different point from the earlier name-reconciliation thread — the concern here is that the .v2 name is wrong for the non-incremental sources this bullet documents.

⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.

@deepakpanda93 deepakpanda93 Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same root cause, same fix — see the reply on the intro line for the detail and the measurements. Corrected in cc34838.

Short version: buildCheckpointFromConfigOverride is not what decides this in the common path. StreamSync.extractCheckpointMetadata uses the Checkpoint object the source returned, and InputBatch(batch, String) wraps a plain string in StreamerCheckpointV2 unconditionally (InputBatch.java:43-44). Running ParquetDFSSource at table version 9 and at table version 6 produced streamer.checkpoint.key.v2 in both cases, so a DFS checkpoint does not stay on the deltastreamer.* key.

The bullet no longer claims a table-version rule:

--checkpoint will set the matching reset key in the commit file to overwrite the current checkpoint, either streamer.checkpoint.reset.key.v2 or deltastreamer.checkpoint.reset_key.

Worth keeping in mind that the pre-existing text said streamer.checkpoint.reset_key, which is not a key Hudi writes at all — so this bullet needed correcting regardless of which rule governs the choice.


#### Resetting the checkpoint for the Hudi incremental source

When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by
**completion time** instead of requested instant time, and records it in the commit metadata under
`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which
one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint`
takes a bare completion time.

```shell

@hudi-agent hudi-agent Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 This section could clarify the 1.0.0 behavior: it says on 1.0.0 --checkpoint "takes a bare completion time," but the PR's own reproduction states the bare value fed to every version was the instant's requested time, and that value was accepted on 1.0.0. That suggests 1.0.0 interprets a bare --checkpoint as a requested instant time (consistent with #12718's note that 1.x incremental sources were still request-time based), not a completion time. Could you double-check whether 1.0.0 treats the bare value as requested or completion time? @yihua could you confirm the pre-1.0.1 checkpoint semantics here?

⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.

@deepakpanda93 deepakpanda93 Aug 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good challenge — I re-tested rather than argue from the earlier run, and the doc statement holds: 1.0.0 reads a bare --checkpoint as a completion time.

The inference in the comment is that because the bare value was accepted on 1.0.0, and that value happened to be a requested instant time, 1.0.0 must interpret bare values as requested times. Acceptance does not imply interpretation, though — 1.0.0 does no validation of the value at all, so it accepts any timestamp-shaped string regardless of which clock it came from. What the value means only shows up in which records get ingested.

So I ran the discriminating pair on 1.0.0. Source instant #2: requested 20260810143709423, completed 20260810143710155.

bare --checkpoint rows ingested instant #2
20260810143709423 (its requested time) 21,22,31,32,41,42 included
20260810143710155 (its completion time) 31,32,41,42 excluded

If a bare value were a requested time, passing instant #2's requested time would have resumed after instant #2 and excluded its records. It did not. Passing instant #2's completion time is what lands the checkpoint exactly after it. That is completion time semantics.

This also explains the original matrix row that prompted the question. The bare value I fed every version was instant #2's requested time, which is earlier than its completion time, so read as a completion time it sits mid-way and instant #2 is still ahead of the checkpoint — hence six records on 1.0.0 where 1.0.1+ ingests four.

The source agrees. In release-1.0.0, StreamerCheckpointUtils L67-68:

checkpoint = Option.of(CheckpointUtils.targetCheckpointV2(writeTableVersion)
    ? new StreamerCheckpointV2(streamerConfig.checkpoint) : new StreamerCheckpointV1(streamerConfig.checkpoint));

At table version 8 targetCheckpointV2 is true, so the raw string is wrapped in StreamerCheckpointV2 — a completion time — with no translation and no parsing. HoodieIncrSourceCheckpointValUtils, which is what introduces the prefixes and their validation, does not exist in 1.0.0 (404 at that tag).

On #12718's note that 1.x was "still using request time based handling": that was written in January 2025 and is now stale, which the PR body covers. The 1.0.0 run with no override stores streamer.checkpoint.key.v2 holding a completion time, so even 1.0.0 was already completion time based in storage — it just had no way to disambiguate an override, which is the gap #12718 closed.

No change made for this one. @yihua a confirmation of the pre-1.0.1 semantics would still be welcome.

# resume after the instant with this requested instant time
--checkpoint resumeFromInstantRequestTime:20250110120000000

# resume after the instant with this completion time
--checkpoint resumeFromInstantCompletionTime:20250110120005000
```

Both forms resume from the same position. The request time form is translated internally to the completion time of that
same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim
under `streamer.checkpoint.reset.key.v2`.

Passing a bare timestamp is rejected:

```plain
Illegal checkpoint key override `20250110120000000`. Valid format is either
`resumeFromInstantRequestTime:<checkpoint value>` or `resumeFromInstantCompletionTime:<checkpoint value>`.
```

:::note
`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue
to take a plain `--checkpoint` value regardless of the table version.
:::

:::caution
Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using
a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the
job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later
run.
:::

### Transformers

Hudi Streamer supports custom transformation on records before writing to storage. This is done by supplying
Expand Down
45 changes: 43 additions & 2 deletions website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,14 +310,55 @@ Read more in depth about concurrency control in the [concurrency control concept
`HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under
`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies
depends on the source, and for the Hudi incremental source also on the table version — see below.

If you need to change the checkpoints for reprocessing or replaying data you can use the following options:

- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.

#### Resetting the checkpoint for the Hudi incremental source

When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by
**completion time** instead of requested instant time, and records it in the commit metadata under
`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which
one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint`
takes a bare completion time.

```shell
# resume after the instant with this requested instant time
--checkpoint resumeFromInstantRequestTime:20250110120000000

# resume after the instant with this completion time
--checkpoint resumeFromInstantCompletionTime:20250110120005000
```

Both forms resume from the same position. The request time form is translated internally to the completion time of that
same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim
under `streamer.checkpoint.reset.key.v2`.

Passing a bare timestamp is rejected:

```plain
Illegal checkpoint key override `20250110120000000`. Valid format is either
`resumeFromInstantRequestTime:<checkpoint value>` or `resumeFromInstantCompletionTime:<checkpoint value>`.
```

:::note
`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue
to take a plain `--checkpoint` value regardless of the table version.
:::

:::caution
Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using
a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the
job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later
run.
:::

### Transformers

`HoodieStreamer` supports custom transformation on records before writing to storage. This is done by supplying
Expand Down
45 changes: 43 additions & 2 deletions website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,14 +310,55 @@ Read more in depth about concurrency control in the [concurrency control concept
`HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under
`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies
depends on the source, and for the Hudi incremental source also on the table version — see below.

If you need to change the checkpoints for reprocessing or replaying data you can use the following options:

- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.

#### Resetting the checkpoint for the Hudi incremental source

When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by
**completion time** instead of requested instant time, and records it in the commit metadata under
`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which
one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint`
takes a bare completion time.

```shell
# resume after the instant with this requested instant time
--checkpoint resumeFromInstantRequestTime:20250110120000000

# resume after the instant with this completion time
--checkpoint resumeFromInstantCompletionTime:20250110120005000
```

Both forms resume from the same position. The request time form is translated internally to the completion time of that
same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim
under `streamer.checkpoint.reset.key.v2`.

Passing a bare timestamp is rejected:

```plain
Illegal checkpoint key override `20250110120000000`. Valid format is either
`resumeFromInstantRequestTime:<checkpoint value>` or `resumeFromInstantCompletionTime:<checkpoint value>`.
```

:::note
`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue
to take a plain `--checkpoint` value regardless of the table version.
:::

:::caution
Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using
a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the
job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later
run.
:::

### Transformers

`HoodieStreamer` supports custom transformation on records before writing to storage. This is done by supplying
Expand Down
45 changes: 43 additions & 2 deletions website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,14 +310,55 @@ Read more in depth about concurrency control in the [concurrency control concept
Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
Checkpoints are saved in the .hoodie commit file. Completion time checkpoints are stored under
`streamer.checkpoint.key.v2` and request time checkpoints under `deltastreamer.checkpoint.key`. Which one applies
depends on the source, and for the Hudi incremental source also on the table version — see below.

If you need to change the checkpoints for reprocessing or replaying data you can use the following options:

- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
- `--checkpoint` will set the matching reset key in the commit file to overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or `deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset.
- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.

#### Resetting the checkpoint for the Hudi incremental source

When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by
**completion time** instead of requested instant time, and records it in the commit metadata under
`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which
one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint`
takes a bare completion time.

```shell
# resume after the instant with this requested instant time
--checkpoint resumeFromInstantRequestTime:20250110120000000

# resume after the instant with this completion time
--checkpoint resumeFromInstantCompletionTime:20250110120005000
```

Both forms resume from the same position. The request time form is translated internally to the completion time of that
same instant, and ingestion proceeds in completion time order either way. Whichever value you pass is recorded verbatim
under `streamer.checkpoint.reset.key.v2`.

Passing a bare timestamp is rejected:

```plain
Illegal checkpoint key override `20250110120000000`. Valid format is either
`resumeFromInstantRequestTime:<checkpoint value>` or `resumeFromInstantCompletionTime:<checkpoint value>`.
```

:::note
`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from completion time checkpoints. They continue
to take a plain `--checkpoint` value regardless of the table version.
:::

:::caution
Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades or downgrades the table version while using
a Hudi incremental source. Rather than risk applying the override under the wrong checkpoint semantics, Hudi fails the
job with a message asking you to drop those options. Let the upgrade finish first, then reset the checkpoint on a later
run.
:::

### Transformers

Hudi Streamer supports custom transformation on records before writing to storage. This is done by supplying
Expand Down
Loading