I'm trying to develop a new DAG on the analytics airflow instance to automate the differential privacy pageview release.
In order to import custom package into my dev airflow instance, I added a new config to artifacts.yaml (which uses an outside URL to get the package from Gitlab). However, on running ./run_dev_instances.sh, I get the following error:
DAG Import Errors (1)
Broken DAG: [/srv/home/htriedman/airflow-dags/analytics/dags/differential_privacy/country_project_page_daily.py] Traceback (most recent call last):
File "/srv/home/htriedman/airflow-dags/wmf_airflow_common/operators/spark.py", line 270, in for_virtualenv
**kwargs
File "/srv/home/htriedman/airflow-dags/wmf_airflow_common/hooks/spark.py", line 500, in kwargs_for_virtualenv
f'virtualenv_archive must be a .zip, .tar.gz or .tgz file. Was {virtualenv_archive}'
ValueError: virtualenv_archive must be a .zip, .tar.gz or .tgz file. Was hdfs:///wmf/cache/artifacts/airflow/analytics/https___gitlab.wikimedia.org_repos_security_differential-privacy_-_package_files_716_downloadWhat background do I need to understand why this doesn't work? What could I change here to fix this bug? Thanks so much.