Avoid data race in grpc alarm

[windtalker]

What problem does this PR solve?

Issue Number: close #10107

Problem Summary:

In current implementation, Alarm is hold by MPPGatherTaskSet, and in MPPTaskManager::abortMPPGather, all the Alarm will be deconstructed since it call gather_task_set->alarms.clear();
But when gather_task_set->alarms.clear(); is called, the EstablishCallData may still inside grpc's core, and it hold a raw pointer of AlarmImpl, although inside AlarmImpl, it use atomic to try to make it thread-safe

 void Ref() { gpr_ref(&refs_); }
  void Unref() {
    if (gpr_unref(&refs_)) {
      delete this;
    }
  }

But Unref/Ref is not thread safe because in grpc's implementation, if EstablishCallData is put back to grpc's core, it does not call Ref immediately, instead, looks like grpc only call Ref if some event happens:

 void Set(::grpc::CompletionQueue* cq, gpr_timespec deadline, void* tag) {
    grpc_core::ApplicationCallbackExecCtx callback_exec_ctx;
    grpc_core::ExecCtx exec_ctx;
    GRPC_CQ_INTERNAL_REF(cq->cq(), "alarm");
    cq_ = cq->cq();
    tag_ = tag;
    GPR_ASSERT(grpc_cq_begin_op(cq_, this));
    GRPC_CLOSURE_INIT(
        &on_alarm_,
        [](void* arg, grpc_error_handle error) {
          // queue the op on the completion queue
          AlarmImpl* alarm = static_cast<AlarmImpl*>(arg);
          alarm->Ref();
          // Preserve the cq and reset the cq_ so that the alarm
          // can be reset when the alarm tag is delivered.
          grpc_completion_queue* cq = alarm->cq_;
          alarm->cq_ = nullptr;
          grpc_cq_end_op(
              cq, alarm, error,
              [](void* /*arg*/, grpc_cq_completion* /*completion*/) {}, arg,
              &alarm->completion_);
          GRPC_CQ_INTERNAL_UNREF(cq, "alarm");
        },
        this, grpc_schedule_on_exec_ctx);
    grpc_timer_init(&timer_, grpc_timespec_to_millis_round_up(deadline),
                    &on_alarm_);
  }

So there is a case that 2 threads try to delete the AlarmImpl concurrently.

Time	thread 1	thread 2
1	call gpr_unref(&refs_)
2	gpr_unref(&refs_) return true(refs is 0)
3		call Ref()
4	call delete this
5		call Unref(), and it will also try to delete this

What is changed and how it works?

This pr let EstablishCallData to hold the alarm, so it will never be constructed when EstablishCallData is inside grpc's core.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[windtalker]

/run-unit-tests

[windtalker]

/test pull-unit-test

[gengliqi]

Better to use a reference instead of a pointer because this pointer is always not null.

[windtalker]

Can't use reference here because we need to store it in a std::unordered_map in MPPTaskManager

[gengliqi]

std::reference_wrapper can be used to store a reference in unordered_map

[windtalker]

Done

[windtalker]

/test pull-integration-test

[windtalker]

/test pull-integration-test

[windtalker]

/test pull-integration-test

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gengliqi, guo-shaoge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [gengliqi,guo-shaoge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-04-22 08:42:57.793074866 +0000 UTC m=+345121.604865248: ☑️ agreed by gengliqi.
• 2025-04-22 09:06:08.635341513 +0000 UTC m=+346512.447131894: ☑️ agreed by guo-shaoge.

[JaySon-Huang]

/test pull-integration-test

[solotzg]

/run-check-issue-triage-complete

[solotzg]

/cherry-pick release-8.5

[ti-chi-bot]

@solotzg: new pull request created to branch release-8.5: #11030.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.