Page MenuHomePhabricator
Paste P95856

debugging airflow.sdk.execution_time.execute_workload job launcher
ActivePublic
Actions

Authored by atsuko on Aug 3 2026, 11:01 AM.
Project Tags
None
Referenced Files
F97006007: debugging airflow.sdk.execution_time.execute_workload job launcher
Aug 3 2026, 11:01 AM
Subscribers
None
atsuko@deploy1003:~$ kubectl get pod | grep operator-run-cat-os | awk '{print $1}' | xargs -r kubectl describe pod
Name: k8s-pod-operator-run-cat-os-release-8eiju3ys
Namespace: airflow-dev
Priority: -100
Priority Class Name: low-priority-pod
Service Account: airflow-dev-atsuko
Node: dse-k8s-worker1027.eqiad.wmnet/10.64.175.2
Start Time: Mon, 03 Aug 2026 10:58:39 +0000
Labels: airflow-worker=1
airflow_version=3.3.0
app=airflow
component=task-pod
dag_id=k8s_pod_operator
kubeapi_enabled=True
kubernetes_executor=True
release=dev-atsuko
routed_via=dev-atsuko
run_id=manual__2026-08-03T105242.0671750000-3d6cf980f
task_id=run-cat-os-release
try_number=4
Annotations: cni.projectcalico.org/containerID: 7bc203479174dc029930474e07ee58988317409741551f5ef3527318ca654416
cni.projectcalico.org/podIP: 10.67.29.149/32
cni.projectcalico.org/podIPs: 10.67.29.149/32,2620:0:861:302:5ec0:7572:6e16:9595/128
dag_id: k8s_pod_operator
run_id: manual__2026-08-03T10:52:42.067175+00:00
task_id: run-cat-os-release
try_number: 4
Status: Running
IP: 10.67.29.149
IPs:
IP: 10.67.29.149
IP: 2620:0:861:302:5ec0:7572:6e16:9595
Containers:
base:
Container ID: containerd://5ccb3ed39a172765b81fcea7a36d0f4dc8486b5a8d328f394877d32cf89769b6
Image: docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.0-py3.11-2026-08-03-103538-c12a5a673b5b604870bf1e52b7f197caed5ac09c@sha256:490a14ce0283c82dfd9c647db0dfa7415195540179f1ebb6d535f3aba2e73879
Image ID: docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags@sha256:490a14ce0283c82dfd9c647db0dfa7415195540179f1ebb6d535f3aba2e73879
Port: <none>
Host Port: <none>
SeccompProfile: RuntimeDefault
Args:
python
-m
airflow.sdk.execution_time.execute_workload
--json-string
{"token":"eyJhbGciOiJIUzUxMiIsImtpZCI6Im5vdC11c2VkIiwidHlwIjoiSldUIn0.eyJzdWIiOiIwMTlmYzc0Ni05NDVhLTczZWYtYjMxZi0wYjRhZGYxZWFmODQiLCJzY29wZSI6Indvcmtsb2FkIiwianRpIjoiZTlmN2Q5YTA0YjNlNDU0OTkwNjI3YWMwMTFkZjBkZDciLCJhdWQiOiJ1cm46YWlyZmxvdy5hcGFjaGUub3JnOnRhc2siLCJuYmYiOjE3ODU3NTQ3MTksImV4cCI6MTc4NTc1NTMxOSwiaWF0IjoxNzg1NzU0NzE5fQ.UCoQb4uhZXY6ZJrRaXkJe4lT6z8mTDbIgoyy6acPI_zzSTizZR71bKeq6J_f9xjcN3rWhN6BzkhVeTe0i5l2wg","dag_rel_path":"dags/kubernetes_pod_operator_dag.py","bundle_info":{"name":"dags-folder","version":null},"log_path":"dag_id=k8s_pod_operator/run_id=manual__2026-08-03T10:52:42.067175+00:00/task_id=run-cat-os-release/attempt=4.log","ti":{"id":"019fc746-945a-73ef-b31f-0b4adf1eaf84","task_id":"run-cat-os-release","dag_id":"k8s_pod_operator","run_id":"manual__2026-08-03T10:52:42.067175+00:00","try_number":4,"dag_version_id":"019fc743-967b-7fa3-b979-c727fa95fa48","map_index":-1,"hostname":"","context_carrier":{},"queue":"default","pool_slots":1,"priority_weight":1},"sentry_integration":"","type":"ExecuteTask"}
State: Running
Started: Mon, 03 Aug 2026 10:58:45 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 400m
memory: 512Mi
Requests:
cpu: 200m
memory: 256Mi
Environment:
SERVICE_IDENTIFIER: airflow-dev-atsuko
AIRFLOW_HOME: /opt/airflow
GROWTHBOOK_LDAP_SYNC_API_KEY: <set to the key 'GROWTHBOOK_LDAP_SYNC_API_KEY' in secret 'airflow-dev-atsuko-secret-config'> Optional: false
GROWTHBOOK_NEXT_LDAP_SYNC_API_KEY: <set to the key 'GROWTHBOOK_NEXT_LDAP_SYNC_API_KEY' in secret 'airflow-dev-atsuko-secret-config'> Optional: false
AIRFLOW__CORE__FERNET_KEY: <set to the key 'airflow__core__fernet_key' in secret 'airflow-dev-atsuko-secret-config'> Optional: false
AIRFLOW__WEBSERVER__SECRET_KEY: <set to the key 'airflow__webserver__secret_key' in secret 'airflow-dev-atsuko-secret-config'> Optional: false
PG_HOST: <set to the key 'host' in secret 'postgresql-airflow-dev-app'> Optional: false
PG_URI: <set to the key 'uri' in secret 'postgresql-airflow-dev-app'> Optional: false
POOLER_NAME: postgresql-airflow-dev-pooler-rw
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED: False
AIRFLOW_INSTANCE_NAME: test-k8s
AIRFLOW_ENVIRONMENT_NAME: dev_wmf
AIRFLOW_SCHEDULER_HOSTNAME: airflow-scheduler
KRB5CCNAME: /tmp/airflow_krb5_ccache/krb5cc
KRB5_CONFIG: /etc/krb5.conf
AIRFLOW_APPOWNER: atsuko
KRB5_PRINCIPAL: atsuko@WIKIMEDIA
REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED
AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED
SCARF_ANALYTICS: False
ENVOY_SERVICE_NAME: airflow-dev-atsuko-envoy
HADOOP_CONF_DIR: /etc/hadoop/conf
HIVE_CONF_DIR: /etc/hadoop/conf
ARROW_LIBHDFS_DIR: /usr/lib/hadoop/lib/native
LD_LIBRARY_PATH: /usr/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/usr/lib/hadoop/lib/native
SPARK_CONF_DIR: /etc/spark3/conf
KRB5CCNAME: /tmp/airflow_krb5_ccache/krb5cc
KRB5_CONFIG: /etc/krb5.conf
KRB5_KEYTAB: /etc/kerberos/keytabs/airflow.keytab
AIRFLOW_IS_K8S_EXECUTOR_POD: True
Mounts:
/etc/hadoop/conf from airflow-dev-atsuko-hadoop-configuration (rw)
/etc/kerberos/keytabs from airflow-dev-atsuko-kerberos-keytab (ro)
/etc/krb5.conf from airflow-dev-atsuko-kerberos-client-config (rw,path="krb5.conf")
/etc/spark3/conf from airflow-dev-atsuko-spark-configuration (rw)
/opt/airflow/airflow.cfg from airflow-dev-atsuko-config (rw,path="airflow.cfg")
/opt/airflow/airflow_local_settings.py from airflow-dev-atsuko-webserver-config (rw,path="airflow_local_settings.py")
/opt/airflow/airflowfilter from airflow-dev-atsuko-import-config (rw,path="airflowfilter")
/opt/airflow/dags from airflow-dev-atsuko-dags (ro)
/opt/airflow/import_helper.py from airflow-dev-atsuko-import-config (rw,path="import_helper.py")
/opt/airflow/logs from airflow-dev-atsuko-logs (rw)
/opt/airflow/pod_templates from airflow-dev-atsuko-kubernetes-pod-templates (rw)
/opt/airflow/secrets from airflow-dev-atsuko-connections-variables (rw)
/opt/airflow/usr/bin from airflow-dev-atsuko-bash-executables (rw)
/opt/airflow/webserver_config.py from airflow-dev-atsuko-webserver-config (rw,path="webserver_config.py")
/tmp/airflow_krb5_ccache from airflow-dev-atsuko-kerberos-token (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l7g9d (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
airflow-dev-atsuko-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-dev-atsuko-config
Optional: false
airflow-dev-atsuko-webserver-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-dev-atsuko-webserver-config
Optional: false
airflow-dev-atsuko-import-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-dev-atsuko-import-config
Optional: false
airflow-dev-atsuko-bash-executables:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-dev-atsuko-bash-executables
Optional: false
airflow-dev-atsuko-connections-variables:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-dev-atsuko-connections-variables
Optional: false
airflow-dev-atsuko-kubernetes-pod-templates:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-dev-atsuko-kubernetes-pod-templates
Optional: false
airflow-dev-atsuko-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
airflow-dev-atsuko-dags:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: airflow-dev-atsuko-dags-pvc
ReadOnly: false
airflow-dev-atsuko-hadoop-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-dev-atsuko-hadoop-configuration
Optional: false
airflow-dev-atsuko-spark-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-dev-atsuko-spark-configuration
Optional: false
airflow-dev-atsuko-kerberos-client-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-dev-atsuko-kerberos-client-config
Optional: false
airflow-dev-atsuko-kerberos-token:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: airflow-dev-atsuko-kerberos-token-pvc
ReadOnly: false
airflow-dev-atsuko-kerberos-keytab:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-dev-atsuko-kerberos-keytab
Optional: false
kube-api-access-l7g9d:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12s default-scheduler Successfully assigned airflow-dev/k8s-pod-operator-run-cat-os-release-8eiju3ys to dse-k8s-worker1027.eqiad.wmnet
Normal Pulled 6s kubelet Container image "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.0-py3.11-2026-08-03-103538-c12a5a673b5b604870bf1e52b7f197caed5ac09c@sha256:490a14ce0283c82dfd9c647db0dfa7415195540179f1ebb6d535f3aba2e73879" already present on machine
Normal Created 6s kubelet Created container: base
Normal Started 6s kubelet Started container base
atsuko@deploy1003:~$ kubectl get pod | grep operator-run-cat-os | awk '{print $1}' | xargs -r kubectl logs -f
/home/app/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/connection_wrapper.py:32 DeprecatedImportWarning: The `airflow.utils.types.NOTSET` attribute is deprecated. Please use `'airflow.serialization.definitions.notset.ArgNotSet'`.
/home/app/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/connection_wrapper.py:32 DeprecatedImportWarning: The `airflow.utils.types.ArgNotSet` attribute is deprecated. Please use `'airflow.serialization.definitions.notset.ArgNotSet'`.
2026-08-03T10:58:49.495879Z [info ] Successfully loaded custom StatsD client [airflow.sdk.observability.metrics.statsd_logger] loc=statsd_logger.py:45
2026-08-03T10:58:49.667344Z [warning ] The secret_key option in [webserver] has been moved to the secret_key option in [api] - the old setting has been used, but please update your config. [py.warnings] category=DeprecationWarning filename=/home/app/.local/lib/python3.11/site-packages/airflow/configuration.py lineno=446
{"timestamp":"2026-08-03T10:58:50.265217Z","level":"info","event":"Executing workload","workload":"ExecuteTask(dag_rel_path=PurePosixPath('dags/kubernetes_pod_operator_dag.py'), bundle_info=BundleInfo(name='dags-folder', version=None), log_path='dag_id=k8s_pod_operator/run_id=manual__2026-08-03T10:52:42.067175+00:00/task_id=run-cat-os-release/attempt=4.log', ti=TaskInstanceDTO(id=UUID('019fc746-945a-73ef-b31f-0b4adf1eaf84'), task_id='run-cat-os-release', dag_id='k8s_pod_operator', run_id='manual__2026-08-03T10:52:42.067175+00:00', try_number=4, dag_version_id=UUID('019fc743-967b-7fa3-b979-c727fa95fa48'), map_index=-1, hostname='', context_carrier={}, queue='default', pool_slots=1, priority_weight=1, external_executor_id=None, executor_config=None), sentry_integration='', type='ExecuteTask')","logger":"__main__","filename":"execute_workload.py","lineno":51}
{"timestamp":"2026-08-03T10:58:50.266533Z","level":"warning","event":"The base_url option in [webserver] has been moved to the base_url option in [api] - the old setting has been used, but please update your config.","category":"DeprecationWarning","filename":"/home/app/.local/lib/python3.11/site-packages/airflow/executors/base_executor.py","lineno":57,"logger":"py.warnings"}
{"timestamp":"2026-08-03T10:58:50.374754Z","level":"info","event":"Using Public CAs from certifi","logger":"airflow.sdk.api.client","filename":"client.py","lineno":1157}
{"timestamp":"2026-08-03T10:58:51.584671Z","level":"warning","event":"Skipping masking for a secret as it's too short (<5 chars)","logger":"airflow._shared.secrets_masker.secrets_masker","filename":"secrets_masker.py","lineno":599}
{"timestamp":"2026-08-03T10:58:51.586619Z","level":"info","event":"Secrets backends loaded for worker","count":2,"backend_classes":["LocalFilesystemBackend","EnvironmentVariablesBackend"],"logger":"supervisor","filename":"supervisor.py","lineno":2515}
{"timestamp":"2026-08-03T10:58:51.666374Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":42}
{"timestamp":"2026-08-03T10:58:52.667968Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":42}
{"timestamp":"2026-08-03T10:58:54.189922Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":42}
{"timestamp":"2026-08-03T10:58:55.675245Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":42}
{"timestamp":"2026-08-03T10:58:51.669927Z","level":"info","event":"::group::Pre Execute","logger":"task","filename":"task_runner.py","lineno":2303}
{"timestamp":"2026-08-03T10:59:02.244132Z","level":"info","event":"Process exited","pid":57,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":1024}
{"timestamp":"2026-08-03T10:59:02.244539Z","level":"critical","event":"Unhandled exception","logger":"unhandled_exception","filename":"structlog.py","lineno":789,"exception":[{"exc_type":"ConnectError","exc_value":"[Errno 111] Connection refused","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"<frozen runpy>","lineno":198,"name":"_run_module_as_main"},{"filename":"<frozen runpy>","lineno":88,"name":"_run_code"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/execute_workload.py","lineno":106,"name":"<module>"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/execute_workload.py","lineno":102,"name":"main"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/execute_workload.py","lineno":53,"name":"execute_workload"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/executors/base_executor.py","lineno":703,"name":"run_workload"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py","lineno":2524,"name":"supervise_task"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/coordinator.py","lineno":143,"name":"execute_task"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py","lineno":1337,"name":"start"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py","lineno":1359,"name":"_on_child_started"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/api/client.py","lineno":255,"name":"start"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_client.py","lineno":1218,"name":"patch"},{"filename":"/home/app/.local/lib/python3.11/site-packages/tenacity/__init__.py","lineno":338,"name":"wrapped_f"},{"filename":"/home/app/.local/lib/python3.11/site-packages/tenacity/__init__.py","lineno":477,"name":"__call__"},{"filename":"/home/app/.local/lib/python3.11/site-packages/tenacity/__init__.py","lineno":378,"name":"iter"},{"filename":"/home/app/.local/lib/python3.11/site-packages/tenacity/__init__.py","lineno":420,"name":"exc_check"},{"filename":"/home/app/.local/lib/python3.11/site-packages/tenacity/__init__.py","lineno":187,"name":"reraise"},{"filename":"/usr/lib/python3.11/concurrent/futures/_base.py","lineno":449,"name":"result"},{"filename":"/usr/lib/python3.11/concurrent/futures/_base.py","lineno":401,"name":"__get_result"},{"filename":"/home/app/.local/lib/python3.11/site-packages/tenacity/__init__.py","lineno":480,"name":"__call__"},{"filename":"/home/app/.local/lib/python3.11/site-packages/airflow/sdk/api/client.py","lineno":1221,"name":"request"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_client.py","lineno":825,"name":"request"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_client.py","lineno":914,"name":"send"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_client.py","lineno":942,"name":"_send_handling_auth"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_client.py","lineno":979,"name":"_send_handling_redirects"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_client.py","lineno":1014,"name":"_send_single_request"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_transports/default.py","lineno":249,"name":"handle_request"},{"filename":"/usr/lib/python3.11/contextlib.py","lineno":155,"name":"__exit__"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_transports/default.py","lineno":118,"name":"map_httpcore_exceptions"}],"is_group":false,"exceptions":[]},{"exc_type":"ConnectError","exc_value":"[Errno 111] Connection refused","exc_notes":[],"syntax_error":null,"is_cause":true,"frames":[{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_transports/default.py","lineno":101,"name":"map_httpcore_exceptions"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpx/_transports/default.py","lineno":250,"name":"handle_request"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py","lineno":256,"name":"handle_request"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py","lineno":236,"name":"handle_request"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpcore/_sync/connection.py","lineno":101,"name":"handle_request"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpcore/_sync/connection.py","lineno":78,"name":"handle_request"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpcore/_sync/connection.py","lineno":124,"name":"_connect"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpcore/_backends/sync.py","lineno":207,"name":"connect_tcp"},{"filename":"/usr/lib/python3.11/contextlib.py","lineno":155,"name":"__exit__"},{"filename":"/home/app/.local/lib/python3.11/site-packages/httpcore/_exceptions.py","lineno":14,"name":"map_exceptions"}],"is_group":false,"exceptions":[]}]}
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits