KubernetesPodOperator: XCom sidecar cleanup fails with "Permission denied" on kill, causing task failure after successful XCom read

[vinay-kumar-uppala]

Under which category would you file this issue?

Helm chart

Apache Airflow version

3.2.2

What happened and how to reproduce it?

When using KubernetesPodOperator with do_xcom_push=True, the XCom sidecar
container starts correctly and the XCom value is read successfully from
/airflow/xcom/return.json (confirmed via cat in the task logs). However,
immediately after the read, Airflow's extract_xcom_kill step attempts to
terminate the sidecar's idle loop process via kill -2 over kubectl exec,
and this consistently fails with Permission denied:

INFO - The xcom sidecar container has started.
INFO - Running command... if [ -s /airflow/xcom/return.json ]; then cat /airflow/xcom/return.json; else echo __airflow_xcom_result_empty__; fi
INFO - Running command... kill -2 $(pgrep -u $(id -u) -f 'sh')
WARNING - stderr from command: sh: can't kill pid 314: Permission denied
INFO - Primary kill command failed, trying fallback command
INFO - Running command... for f in /proc/[0-9]*/comm; do [ -O $f ] && read c < $f && [ "$c" = "sh" ] && pid=${f%/comm} && kill -2 ${pid##*/}; done
WARNING - stderr from command: sh: can't kill pid 314: Permission denied
INFO - Deleting pod: push-xcom-pod-6jcmp6lx
ERROR - Task failed with exception

Both the primary kill command and the documented fallback command fail
identically. Because the exception is raised inside extract_xcom()
after the value has already been read from the file but before it is
returned to the caller, the task is marked as Failed even though the XCom
payload was technically retrieved from the pod. As a result:

• The XCom value is never persisted to the metadata DB.
• The task instance shows Failed.
• Downstream tasks depending on this task are marked Upstream Failed and
  never run.

Both the primary kill command and the documented fallback command fail
identically. Because the exception is raised inside extract_xcom()
after the value has already been read from the file but before it is
returned to the caller, the task is marked as Failed even though the XCom
payload was technically retrieved from the pod. As a result:

• The XCom value is never persisted to the metadata DB.
• The task instance shows Failed.
• Downstream tasks depending on this task are marked Upstream Failed and
  never run.
  Full traceback:

AirflowException: Pod push-xcom-pod-6jcmp6lx returned a failure.
...
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 770, in execute_sync
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 1151, in post_complete_action
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 1224, in cleanup
PodCommandException: Command failed with stderr: sh: can't kill pid 314: Permission denied
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 763, in execute_sync
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 677, in extract_xcom
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 974, in extract_xcom
File ".../tenacity/__init__.py", line 331, in wrapped_f
File ".../tenacity/__init__.py", line 470, in __call__
File ".../tenacity/__init__.py", line 371, in iter
File ".../tenacity/__init__.py", line 393, in <lambda>
File ".../concurrent/futures/_base.py", line 449, in result
File ".../concurrent/futures/_base.py", line 401, in __get_result
File ".../tenacity/__init__.py", line 473, in __call__
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 1046, in extract_xcom_kill
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 1063, in _exec_pod_command
PodCommandException: Command failed with stderr: sh: can't kill pid 314: Permission denied
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 1043, in extract_xcom_kill
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 1063, in _exec_pod_command

Minimal DAG (see attached file, xcom_kpo_repro_dag.py):

from __future__ import annotations
from datetime import datetime
 
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
 
with DAG(
    dag_id="xcom_kpo_repro",
    schedule=None,
    catchup=False,
) as dag:
    push_xcom_task = KubernetesPodOperator(
        task_id="push_xcom_task",
        name="push-xcom-pod",
        namespace="<namespace>",
        image="python:3.11-slim",
        image_pull_policy="IfNotPresent",
        cmds=["python", "-c"],
        arguments=[
            "import json; "
            "data = {'message': 'hello from the pod', 'count': 42}; "
            "f = open('/airflow/xcom/return.json', 'w'); "
            "json.dump(data, f); "
            "f.close()"
        ],
        do_xcom_push=True,
        get_logs=True,
        on_finish_action="delete_pod",
        in_cluster=True,
    )

Steps:

1. Deploy the DAG above to a cluster running Airflow 3.2.2 with the
   apache-airflow-providers-cncf-kubernetes provider, on a k3s cluster
   (containerd runtime).
2. Trigger the DAG.
3. Observe the task logs: the XCom value is read successfully via cat
   inside the sidecar, but the subsequent kill -2 cleanup step fails with
   Permission denied, and the task is marked Failed.
   To confirm this is unrelated to XCom content or main-container behavior, we
   also reproduced it with the base container modified to sleep() after
   writing the XCom file (to keep the pod alive), then manually ran:

kubectl exec -n <namespace> <pod-name> -c airflow-xcom-sidecar -- sh -c "id; ps aux; kill -2 1"

against the live sidecar container — same Permission denied result outside
of Airflow's own retry/exec logic, confirming it is not specific to
Airflow's particular invocation of the kill command.

What you think should happen instead?

Either:

1. The kill step should succeed against the sidecar's own idle process
   (trap "exit 0" INT; while true; do sleep 1; done;), since it is the
   same container's own PID 1 and no custom securityContext is set on the
   pod (see pod spec below — security_context is None at both the pod
   and container level, as submitted by the operator itself), or
2. If a signal-based teardown is inherently unreliable across container
   runtimes/environments (e.g. due to runtime-level signal isolation between
   an exec session and PID 1), the sidecar teardown should fall back to a
   non-signal-based mechanism (e.g. deleting the pod directly) without
   raising an exception that fails the task, since the XCom value has
   already been successfully extracted at that point in the code path.
   At minimum, a failure in the cleanup/kill step after a successful XCom read
   should not discard the already-retrieved XCom value or fail the task.

Operating System

linux

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

1.22.0 (latest released)

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

• Airflow version: 3.2.2
• Kubernetes distribution: k3s
• Container runtime: containerd (default for k3s)
• Namespace Pod Security Admission label: none set (confirmed via
  kubectl get ns <namespace> -o yaml — no
  pod-security.kubernetes.io/enforce label present)
• Pod-level / container-level securityContext: None in both the
  operator-submitted spec and the live pod spec (no runAsUser,
  runAsNonRoot, capabilities, or seccompProfile set anywhere in the
  submitted spec)
• Sidecar image: alpine:3.23.4 (as injected automatically by the
  operator)
• do_xcom_push: True
• in_cluster: True

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.