[Bug] A race between RocksDB compaction and periodic Raft snapshot save can commit an invalid snapshot directory that contains `__raft_snapshot_meta` but no `data/`

[vaijosh]

Bug Type (问题类型)

others (please edit later)

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

• Server Version: 1.0.0 (Apache Release Version)
• Backend: RocksDB x nodes, HDD or SSD
• OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x
• Data Size: xx vertices, xx edges

Expected & Actual behavior (期望与实际表现)

Expected Behavior

• If snapshot save is triggered while compaction is active (state == doing), snapshot save should return non-OK to JRaft.
• JRaft should not commit a snapshot point unless snapshot data is actually persisted and files are registered.
• Snapshot directories should not be committed in a meta-only state.

Actual Behavior

• During compaction, snapshot handler can exit early without writing data/ (pre-fix path).
• Caller still reports success (done.run(Status.OK())) to JRaft.
• JRaft commits snapshot metadata, so snapshot index advances.
• Snapshot directory may contain only __raft_snapshot_meta, with missing data/.
• On restart/load, snapshot data path is missing, causing load/init failure and retry loops.

Race Condition Timeline (Sequence of Events)

2. dbCompaction() starts for partition P.
3. setState(P, doing) is set, then op.compactRange() begins and blocks.
4. While compaction is running, periodic JRaft snapshot timer fires for P.
5. JRaft invokes PartitionStateMachine.onSnapshotSave(writer, done).
6. PartitionStateMachine calls SnapshotHandler.onSnapshotSave(writer).
7. In vulnerable behavior, handler exits early on busy state (state == doing) without writing data/ and without registering files.
8. Control returns normally, so caller executes done.run(Status.OK()).
9. JRaft treats snapshot save as successful and commits snapshot metadata (__raft_snapshot_meta), advancing snapshot index.
10. Compaction later completes and state changes to compactionDone.
11. A later blank-task snapshot may be triggered, but this is a new snapshot attempt, not a repair of the already committed bad snapshot.
12. If restart/load occurs before a newer valid snapshot supersedes it, loader targets the committed corrupt snapshot and fails on missing data/.## Disk/Log Fingerprint

Corrupt snapshot signature:

• snapshot_<N>/__raft_snapshot_meta exists
• snapshot_<N>/data/ missing
• snapshot_<N>/should_not_load often missing for early-return path

Typical failure path:

• snapshot save appears successful at callback level
• snapshot load later fails because snapshot_<N>/data is absent

Impact

• Affected partition may become unavailable after restart/snapshot load.
• Partition can enter repeated init failure/retry.
• Operational recovery may require manual cleanup and restart if no newer valid snapshot is available.

Severity

High (data-plane availability and recovery risk at partition level).

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

[dosubot]

@vaijosh Good analysis. I can confirm the race condition exists in the current code. Here's what's happening:

In SnapshotHandler.onSnapshotSave, when state == doing (i.e. compaction is in progress), the method silently returns without writing the data/ directory or throwing an exception:

if (state != null && state.get() == BusinessHandler.doing) {
    return;  // ← exits silently without writing data/ or throwing
}

The caller in PartitionStateMachine.onSnapshotSave always calls done.run(Status.OK()) on normal return, since the early return doesn't throw:

snapshotHandler.onSnapshotSave(writer);  // may return early silently
done.run(Status.OK());                   // ← always OK on no exception

The doing state is set in BusinessHandlerImpl.dbCompaction before op.compactRange() blocks, and only transitions to compactionDone after compactRange() completes — so the race window during a long compaction is wide.

Two additional observations that make the failure worse:

• The should_not_load marker is written only at the end of the happy path, so an early-return snapshot directory lacks both data/ and should_not_load.
• On load, onSnapshotLoad calls businessHandler.loadSnapshot unconditionally when should_not_load is absent, which will fail or produce undefined behavior when data/ doesn't exist.
• The deprecated HgSnapshotHandler does not have this early-return path — it always proceeds to write data/ — so this bug is specific to the active SnapshotHandler.

The fix is straightforward: onSnapshotSave should throw an HgStoreException instead of silently returning when state == doing, so PartitionStateMachine calls done.run(new Status(RaftError.EBUSY, ...)) rather than Status.OK(). This causes JRaft to skip committing the snapshot metadata for that round:

if (state != null && state.get() == BusinessHandler.doing) {
    throw new HgStoreException("Snapshot skipped: compaction in progress for group " + groupId);
}

A defensive guard in onSnapshotLoad — checking for the existence of data/ before attempting to load — would also provide a safety net against any corrupt snapshot that slips through.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.