[CONTP-2023] Improve DatadogMonitor reconciliation at scale

[tbavelier]

What does this PR do?

• Adds configurable DatadogMonitor concurrency and status-polling cadence: c6af8bf
• Makes scheduled status polling low priority, keeping create/update/delete work ahead of it: af9c7d5
• Addresses review findings around retry timing and scheduled reconciliation: 42b8d62
• Documents reconciliation priority and retry behavior inline: 5779096

Motivation

Large monitor collections can accumulate polling work and delay lifecycle operations.

Warning

For large DatadogMonitor fleets, we recommend disabling direct operator telemetry with --operatorMetricsEnabled=false (DD_OPERATOR_METRICS_ENABLED=false) for better performance. When enabled (default), every DatadogMonitor starts one metrics-forwarder goroutine.

Reconciliation and Kubernetes Events continue to work, but direct Datadog operator metrics datadog.operator.reconcile.success, datadog.operator.<resource>.custom_resource.count, etc.) and events for DatadogAgent, DatadogAgentInternal, and DatadogMonitor resources are disabled. THESE DO NOT AFFECT Agent telemetry, this is purely self-telemetry from the operator

If operator own telemetry must remain enabled, raise the --maximumGoroutines (DD_MAXIMUM_GOROUTINES) health-check threshold above the expected total goroutine count; its default is 500 and it is not a runtime goroutine limit.

QA

Manual scale and priority check:

1. Deploy the operator with DatadogMonitor enabled and --datadogMonitorMaxConcurrentReconciles=10 (as well as operatorMetricsEnabled=false or maximumGoroutines=10000)
2. Create 1,000 monitors at once with the helper below and verify they receive Datadog monitor IDs.
3. Restart with --datadogMonitorMaxConcurrentReconciles=1 to accumulate scheduled status-polling work.
4. Create one additional monitor and verify it is reconciled ahead of the polling backlog.
5. Restore the worker count and delete the load-test monitors.

DatadogMonitor bulk load helper

#!/usr/bin/env bash
set -euo pipefail

ACTION="${1:-}"
COUNT="${2:-}"
NAMESPACE="${NAMESPACE:-default}"
PREFIX="${PREFIX:-datadogmonitor-load}"

if [[ "$ACTION" != create && "$ACTION" != delete ]] || ! [[ "$COUNT" =~ ^[1-9][0-9]*$ ]]; then
  echo "usage: NAMESPACE=default PREFIX=datadogmonitor-load $0 {create|delete} <count>" >&2
  exit 1
fi

render() {
  local i name
  for ((i = 1; i <= COUNT; i++)); do
    name="${PREFIX}-${i}"
    cat <<EOF
---
apiVersion: datadoghq.com/v1alpha1
kind: DatadogMonitor
metadata:
  name: ${name}
spec:
  name: ${name}
  type: metric alert
  query: avg(last_5m):avg:system.load.1{*} > 100000
  message: DatadogMonitor controller load test
  tags:
    - generated:kubernetes
    - test:datadogmonitor-load
EOF
  done
}

case "$ACTION" in
  create) render | kubectl --namespace "$NAMESPACE" create --filename - ;;
  delete) render | kubectl --namespace "$NAMESPACE" delete --filename - --ignore-not-found ;;
esac

./datadogmonitor-load.sh create 1000
./datadogmonitor-load.sh delete 1000

The full batch is sent through one kubectl create or kubectl delete invocation.

[datadog-datadog-us1-prod]

 

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

• Patch coverage: 69.70% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 69.70%
• Overall Coverage: 50.19% (+0.07%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5779096 | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af9c7d5002

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[tbavelier]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 5779096b20

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".