Productionize the P.O.C. done in T322534.
Implement a NotebookOperator that runs Jupyter notebooks in Airflow using Skein and Papermill.
It should accept as parameters:
- The path to the notebook file itself in HDFS (this can be treated as an Airflow artifact)
- The path to a packaged conda environment in HDFS (.tar.gz file - also can be treated as an Airflow artifact)
- Any parameters passed directly to the notebook via Papermill, these should be dynamic (you might want to pass i.e. {{ execution_date.year }} )