For reference: https://github.com/wikimedia/wikimedia-discovery-analytics/blob/master/airflow/plugins/wmf_airflow/hdfs_cli.py

See https://phabricator.wikimedia.org/T284172#7144227 for an example of how to use pyarrow.fs.HadoopFileSystem to connect to HDFS (I did not find the pyarrow docs obvious).

A PR already exists in airflow to do this. We could reopen it and follow up:

https://github.com/apache/airflow/pull/3560/files

We've been looking into pyarrow + airflow a little bit in general, and I need a place to park some findings, so here it is!

pyarrow is not thread safe, and Airflow's LocalExecutor uses python multiprocessing Queues to execute tasks. The task itself is either executed by via forking (the default) and then calling the relevant airflow function, or by calling subprocess.check_call with an airflow tasks run ... command. I had originally thought that by setting execute_tasks_new_python_interpreter=True, the subprocess.check_call method would be used and we'd avoid the deadlock we sometimes see. It seems this doesn't work either. I believe the fact that multiprocessing is used by LocalExecutor means that we cannot use pyarrow from within the scheduler process at DAG parse time.

We CAN use pyarrow with LocalExecutor inside of tasks. As long as the pyarrow HadoopFileSystem is instantiated inside of the task (i.e. in the forked process, or in the new python process), everything is fine. We get a deadlock if we use pyarrow in the DAG itself and also in tasks. (Strangely we don't seem to get a deadlock if we only use pyarrow in the DAG, but not in tasks...which I don't understand).

So, this means using pyarrow HadoopFileSystem for something like an HDFS Sensor (which is a special type of Operator) should be fine. But using pyarrow to dynamically generate DAGs (like we wanted to do for Refine) won't be possible, at least not with LocalExecutor or SequentialExecutor.

So TBD what to do. If we really want to use pyarrow to generate the Refine DAGs, we'll probably need to use CeleryExecutor. This should be easy, as we'd just run celery on the airflow node, (not with distributed workers). However, it will make testing Refine DAGs hard, as you won't be able to run them inside of a airflow development instance scheduler.