fix(gpu): force disable eBPF probes - #3302
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 764b0d900e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
🎯 Code Coverage (details) 🔗 Commit SHA: d5ed386 | Docs | View more details | Give us feedback! |
| // the privileged host access rather than the eBPF probes. | ||
| disableEBPFProbesEnvVar := &corev1.EnvVar{ | ||
| Name: DDEnableEBPFProbesEnvVar, | ||
| Value: "false", |
There was a problem hiding this comment.
suggestion (non-blocking): this force-disables the probes, which makes the privilegedMode field doc stale. The marker at api/datadoghq/v2alpha1/datadogagent_types.go:721 still reads "PrivilegedMode enables GPU Probe module in System Probe" — accurate for the module, but the probes it implies are now off by default. That comment is the source for both generated docs (docs/configuration_public.md:235, docs/configuration.v2alpha1.md:136), so they drift with it. Update the marker to note the eBPF probes are disabled by default and privileged mode is retained for cgroup permission patching, then run make generate && make manifests so both generated docs move in lockstep.
| // add the env var to the core agent as well, so both containers agree that the | ||
| // eBPF probes are disabled and the core GPU check does not poll a system-probe | ||
| // module that has the probes turned off | ||
| managers.EnvVar().AddEnvVarToContainer(apicommon.CoreAgentContainerName, disableEBPFProbesEnvVar) |
There was a problem hiding this comment.
suggestion (non-blocking): this hard-codes the probes off with no first-class opt-out, which is the right default given the GB300 crashes. Worth documenting the escape hatch that already exists: feature config runs before spec.override, and env merge is last-writer-wins by name (OverrideCurrentEnvVarMergeFunction in merger/envvars.go), so a user who still needs the probes can set DD_GPU_MONITORING_ENABLE_EBPF_PROBES=true via spec.override.nodeAgent.containers and it wins cleanly, with no duplicate env var. A line in the release note / docs saying probes are now disabled by default in privileged mode and can be re-enabled that way would save a support round-trip.
This commit forces the disablement of the eBPF probes when privileged mode is enabled.
Add DD_GPU_MONITORING_ENABLE_EBPF_PROBES=false to the core agent container as well as system-probe, so both containers agree the eBPF probes are disabled. Previously only system-probe received the flag, while the core agent still saw GPU monitoring enabled and would poll a system-probe module whose probes were off, causing recurring refresh warnings and empty process metrics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
f4b5d02 to
d5ed386
Compare
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: done
and removed
mergequeue-status: in_progress
labels
* fix(gpu): force disable eBPF probes This commit forces the disablement of the eBPF probes when privileged mode is enabled. * fix(gpu): disable eBPF probes on core agent too Add DD_GPU_MONITORING_ENABLE_EBPF_PROBES=false to the core agent container as well as system-probe, so both containers agree the eBPF probes are disabled. Previously only system-probe received the flag, while the core agent still saw GPU monitoring enabled and would poll a system-probe module whose probes were off, causing recurring refresh warnings and empty process metrics. * Address PR feedback --------- (cherry picked from commit 2e78244) Co-authored-by: Mark Spicer <mark.spicer@datadoghq.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
5 participants
What does this PR do?
In privileged mode the GPU eBPF probes are now force-disabled (DD_GPU_MONITORING_ENABLE_EBPF_PROBES=false) on both the core agent and system-probe; privileged mode is retained for cgroup permission patching. Clusters that still need the probes can re-enable them by setting DD_GPU_MONITORING_ENABLE_EBPF_PROBES=true on those containers under spec.override.nodeAgent.containers — overrides are applied after feature config and env vars merge last-writer-wins by name, so the override wins with no duplicate env var.
Motivation
The eBPF probes for GPU Monitoring are deprecated. Some customers still have the previously recommended setting gpu.privilegedMode enabled. These probes have caused crashes on NVIDIA GB300 Grace Blackwell Ultra machines. We want to disable the probes by default moving forward so they no longer cause problems. We still need to support
privilegedModein some environments for cgroup permission patching.Additional Notes
We've had two customer cases of this and we want to prevent further cases.
Minimum Agent Versions
Are there minimum versions of the Datadog Agent and/or Cluster Agent required?
Describe your test plan
The additional unit tests.
Checklist
bug,enhancement,refactoring,documentation,tooling, and/ordependenciesqa/skip-qalabel