Page MenuHomePhabricator
Create Task
Maniphest T428099

Airflow Dev Env Task Shell Pod Missing CLASSPATH Environment Variable
Closed, ResolvedPublic
Actions

Description

While testing a very simple DAG with one task that is using pyarrow and fsspec to download a file from HDFS to local filesystem, we encountered the following error:

File "pyarrow/_hdfs.pyx", line 96, in pyarrow._hdfs.HadoopFileSystem.__init__
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: HDFS connection failed

After some debugging, we discovered that the airflow-devenv-username-task-shell-xxxxx pods do not have the CLASSPATH environment variable set:

runuser@airflow-dev-astein-task-shell-b4dbb8586-sd942:/opt/airflow$ export | grep -i classpath
runuser@airflow-dev-astein-task-shell-b4dbb8586-sd942:/opt/airflow$

After manually setting the following, the pyarrow and fsspec code worked as expected:

runuser@airflow-dev-astein-task-shell-b4dbb8586-sd942:/opt/airflow$ export CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath --glob`

Please include this env var in the Airflow's helm chart deployments so it's available to airflow-devenv DAGs and their tasks.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
airflow: export the CLASSPATH environment variable into the task-pod shelloperations/deployment-chartsmaster+138 -0
airflow: upgrade the imageoperations/deployment-chartsmaster+1 -58
airflow: add ARROW_LIBHDFS_DIR/LD_LIBRARY_PATH to the of hadoop env varsoperations/deployment-chartsmaster+9 -2
airflow: export the CLASSPATH environment variable into the task-pod shelloperations/deployment-chartsmaster+138 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
docker: hardcode the CLASSPATH environment variablerepos/data-engineering/airflow-dags!2288brouberolT428099main
docker: export the CLASSPATH environment variables to all containers using the airflow imagerepos/data-engineering/airflow-dags!2283brouberolT428099main
Customize query in GitLab

Event Timeline

amastilovic created this task.Wed, Jun 3, 8:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Jun 3, 8:44 PM
amastilovic updated the task description. (Show Details)Wed, Jun 3, 10:56 PM
AStein-WMF added a comment.Thu, Jun 4, 6:48 PM

some more info- i was able to connect to hdfs and get file info in airflow devenv without CLASSPATH, by using wmf_airflow_common.util.hdfs_client. i dont know if that's the preferred way to interact with hdfs. code now looks like:

@task(executor_config=executor_config_with_proxy)
    def download_from_hdfs(hdfs_url: str) -> str:
        hdfs = util.hdfs_client(hadoop_name_node)
        file_info = hdfs.get_file_info(hdfs_url)
amastilovic added a comment.EditedMon, Jun 8, 10:31 AM

Note that wmf_airflow_common.util.hdfs_client works because it sets the CLASSPATH itself - it would be preferable for this variable to be set in the deployment chart instead:

def hdfs_client(name_node: str) -> HadoopFileSystem:
    """Returns an HDFS client for the given Hadoop name node.
    Args:
        name_node   URI for Hadoop name node.
                    (i.e.: hdfs://analytics-hadoop)
    """
    stream = os.popen("hdfs classpath --glob")
    os.environ["CLASSPATH"] = stream.read().strip()
    return HadoopFileSystem(name_node)
amastilovic claimed this task.Tue, Jun 9, 1:26 PM
Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE.Tue, Jun 9, 1:31 PM
Gehel moved this task from Backlog - project to Quick Wins on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
gerritbot added a comment.Tue, Jun 9, 3:07 PM

Change #1299527 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: export the CLASSPATH environment variable into the task-pod shell

https://gerrit.wikimedia.org/r/1299527

gerritbot added a project: Patch-For-Review.Tue, Jun 9, 3:07 PM
gerritbot added a comment.Tue, Jun 9, 6:12 PM

Change #1299525 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: export the CLASSPATH environment variable into the task-pod shell

https://gerrit.wikimedia.org/r/1299525

gerritbot added a comment.Wed, Jun 10, 6:41 AM

Change #1299525 abandoned by Brouberol:

[operations/deployment-charts@master] airflow: export the CLASSPATH environment variable into the task-pod shell

https://gerrit.wikimedia.org/r/1299525

CodeReviewBot added a comment.Wed, Jun 10, 6:59 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2283

docker: export the CLASSPATH environment variables to all containers using the airflow image

brouberol changed the task status from Open to In Progress.Wed, Jun 10, 7:00 AM
brouberol claimed this task.
brouberol moved this task from Quick Wins to Needs Review on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
CodeReviewBot added a comment.Wed, Jun 10, 11:22 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2283

docker: export the CLASSPATH environment variables to all containers using the airflow image

brouberol added a comment.Wed, Jun 10, 11:34 AM

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/jobs/855777 published

docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-112304-329cb255db7013430d0d66d170db6803f44d855d@sha256:a993deaf913d7443a7d48e6740c265556bb62056a87a5be9165bf70042d8459a

which should contain all required environment variables. @amastilovic can I let you test it in a devenv?

gerritbot added a comment.Wed, Jun 10, 12:56 PM

Change #1300131 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: add ARROW_LIBHDFS_DIR/LD_LIBRARY_PATH to the of hadoop env vars

https://gerrit.wikimedia.org/r/1300131

CodeReviewBot added a comment.Wed, Jun 10, 1:04 PM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2288

docker: hardcode the CLASSPATH environment variable

CodeReviewBot added a comment.Wed, Jun 10, 1:15 PM

amastilovic merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2288

docker: hardcode the CLASSPATH environment variable

gerritbot added a comment.Wed, Jun 10, 1:33 PM

Change #1300131 merged by Brouberol:

[operations/deployment-charts@master] airflow: add ARROW_LIBHDFS_DIR/LD_LIBRARY_PATH to the of hadoop env vars

https://gerrit.wikimedia.org/r/1300131

gerritbot added a comment.Wed, Jun 10, 1:42 PM

Change #1300147 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: upgrade the image

https://gerrit.wikimedia.org/r/1300147

gerritbot added a comment.Wed, Jun 10, 1:58 PM

Change #1300147 merged by Brouberol:

[operations/deployment-charts@master] airflow: upgrade the image

https://gerrit.wikimedia.org/r/1300147

brouberol added a comment.Wed, Jun 10, 2:01 PM

We managed to make it work by hardcoding CLASSPATH in the image and setting the other env vars in the chart.

brouberol added a comment.Wed, Jun 10, 2:22 PM

All airflow instances have been redeployed with the CLASSPATH / LD_LIBRARY_PATH / ARROW_LIBHDFS_DIR env vars. We should be good to close.

brouberol closed this task as Resolved.Wed, Jun 10, 4:59 PM
brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.Fri, Jun 12, 7:18 AM
gerritbot added a comment.Fri, Jun 12, 12:41 PM

Change #1299527 abandoned by Brouberol:

[operations/deployment-charts@master] airflow: export the CLASSPATH environment variable into the task-pod shell

Reason:

Sorry, I should have closed this one. This was made redundant by https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2288 and https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1300131

https://gerrit.wikimedia.org/r/1299527

Maintenance_bot removed a project: Patch-For-Review.Fri, Jun 12, 1:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits