Skip to content

fix: DDL and DML execution order race condition#990

Open
Suarez1024 wants to merge 1 commit into
alibaba:developfrom
Suarez1024:ddl-order-bugfix
Open

fix: DDL and DML execution order race condition#990
Suarez1024 wants to merge 1 commit into
alibaba:developfrom
Suarez1024:ddl-order-bugfix

Conversation

@Suarez1024

Copy link
Copy Markdown

Problem

DDL operations were not guaranteed to execute in the correct order relative to DML operations, causing data inconsistency:

  1. DDL may execute before preceding DML
  2. DDL may execute after succeeding DML

Root Causes

  1. checkCheckpointUpdate waits for checkpoint timestamp, not actual DML completion
  2. DDL and DML execute in parallel across different workers

Solution

DDL bypasses the worker pipeline and is executed synchronously by the Batcher goroutine, ensuring natural serialization with DML.

Files Changed

  • collector/batcher.go
  • collector/syncer.go
  • collector/batcher_test.go
  • docs/fixes/ddl-barrier-race-condition.md

Details

See docs/fixes/ddl-barrier-race-condition.md for full analysis and implementation details.

Fixes #934

@CLAassistant

CLAassistant commented Jul 9, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@zhongli-james

Copy link
Copy Markdown
Collaborator

@Suarez1024 感谢提交 PR 并签署 CLA!我仔细看了实现,结合 #934 的讨论,有几个比较关键的问题需要先对齐,目前不建议按当前方案合并:

  • 架构职责:在 DDL 和DML操作没有按照oplog顺序同步,导致源库和目标库数据不一致 #934 里我提到过「出于职责划分,不建议 batcher 做实际 oplog 回放动作」。这个 PR 的 DDLExecutor 在 batcher goroutine里新开了一条到目标端的连接并同步执行 DDL,正是违背这一点。而且这条连接全程没有 Close(),会造成连接泄漏。
  • 非 direct tunnel 的正确性回归(阻塞性):DDLExecutor 只在 direct tunnel 下创建,其它 tunnel 走 executeDDLDirectly 的 fallback,把 DDL addIntoBatchGroupbatchGroup。但此时 dispatchBatches 已经在前面把 batchGroup 下发过了,Step 2 追加的 DDL 不会再被 dispatch,下一轮BatchMore 会重建 batchGroup——这会导致非 direct tunnel 下 DDL 被静默丢弃,而随后 checkpoint(true, ddlTs) 又把 checkpoint 推过了这条DDL,重启也无法补回。这实际上引入了一类新的、不可恢复的数据不一致。
  • 是否已被 Dev zhongli #981 解决:当前 develop 上,BatchMore 遇到 DDL 会把后续 DML 扣在 remainLogs 中不下发,配合 Dev zhongli #981 已改为无界等待的checkCheckpointUpdate,DML→DDL→DML 的顺序应当已由两段式 barrier 保证。能否先提供一个在当前 develop分支上能稳定复现乱序的失败用例?这样我们能明确这个 race 是否仍然存在,避免重复解决。
  • 如果确实存在 develop 上仍未覆盖的乱序窗口,我更倾向在现有 barrier 机制内加固(例如强化 DDL 单独下发 +
    完成确认),而不是新增一条并行写入路径。

期待你的复现用例,我们再一起定改进方案~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DDL 和DML操作没有按照oplog顺序同步,导致源库和目标库数据不一致

3 participants