Change Details

As part of Airflow's dependency management system, we want to have an Airflow DAG running periodically (every 5 mins?), that grabs all packages in a given Archiva repository and syncs them to a predetermined HDFS location. That will allow teams to reference their dependencies in spark-submit calls with HDFS paths, with no deployment step needed. The only requirement would be to package the dependency and upload it to Archiva.

As part of Airflow's dependency management system, we want to have an automatized process in place, that grabs all packages in a given Archiva repository and syncs them to a predetermined HDFS location. That will allow teams to reference their dependencies in spark-submit calls with HDFS paths, with no deployment step needed. The only requirement would be to package the dependency and upload it to Archiva.

As part of Airflow's dependency management system, we want to have an Airflow DAG running periodically (every 5 mins?)automatized process in place, that grabs all packages in a given Archiva repository and syncs them to a predetermined HDFS location. That will allow teams to reference their dependencies in spark-submit calls with HDFS paths, with no deployment step needed. The only requirement would be to package the dependency and upload it to Archiva.