1 Week in this sprint :)

I believe this spike work is finished.

We have implemented a simple P.O.C. that confirms we can run Jupyter notebooks in Airflow.
Here's the GitLab merge request of the P.O.C., we can use this as a reference if/when we implement this feature.
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/196
I created a task to potentially productionize it: T325185.

For the test we also created a packaged conda environment using conda-pack, that contained the necessary libraries to execute naive jupyter notebooks using papermill.
This was the code used on a stats machine:

export http_proxy="http://webproxy.eqiad.wmnet:8080"
export https_proxy="http://webproxy.eqiad.wmnet:8080"
source /usr/lib/airflow/etc/profile.d/conda.sh
conda create --name notebook_operator_env
conda activate notebook_operator_env
conda install python=3.10
conda install jupyter
conda install -c conda-forge papermill
conda install -c conda-forge conda-pack
conda-pack
conda deactivate

After that, I put it in HDFS under the analytics-privatedata user with proper permissions.
For this part, I also created a new task, to set up a repository for the conda environment generation.
This repository should build PA's packaged conda environments for the execution of Jupyter notebooks via CI.
T325195

We also have looked into the list of notebooks scheduled by the PA team to collect requirements for the feature.
I added a new sheet to the Airflow migration spreadsheet with that information (PA Jobs):
https://docs.google.com/spreadsheets/d/1lfK5Idteh6zPSlCWyH34FJCl_Lcm8401Wm59Jgk-7wM
The conclusions of this analysis are in https://phabricator.wikimedia.org/T322666#8460702

I split part of the initial objectives of this task to a separate task T325181, since we discussed in standup that we'd do this over the course of the next couple weeks.