Customize query in gerrit
Details
Details
Related Changes in Gerrit:
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Intial build of the refinery container image | repos/data-engineering/refinery!1 | btullis | add_refinery_blubber | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | brouberol | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Resolved | brouberol | T364389 Migrate the airflow scheduler components to Kubernetes | |||
| Resolved | bking | T380621 Migrate the airflow-search scheduler to Kubernetes | |||
| Resolved | dcausse | T386097 Re-enable drop_old_data_daily in airflow-search |
Event Timeline
Comment Actions
btullis opened https://gitlab.wikimedia.org/repos/data-engineering/refinery/-/merge_requests/1
Intial build of the refinery container image
Comment Actions
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/refinery/-/merge_requests/1
Intial build of the refinery container image
Comment Actions
By way of a quick update, there were 3 issues discovered post-migration.
- The search team were still using the HiveOperator for one of their DAGs, but our Airflow image does not include the hive CLI.
- Some of the DAGs (e.g. drop_old_data_daily) were using the BashOperator were attempting to run some refinery scripts, specifically refinery-drop-older-than. These scripts are not available in our Airflow image.
- Some of the DAGs that used the SparkOperator were failing and the task logs just showed confusing errors that the jobs had been killed.
We have largely addressed all of these now, although 2 won't be fixed until next week, at the earliest.
Fixes:
- @dcausse switched the popularity_score DAG to spark-sql
- We have decided to T383417: Create a container image for analytics/refinery to be used with Airflow tasks and launch this with the KubernetesPodOperator to do these tasks
- We ascertained that the issue was primarily caused by the use of cluster mode when launching the spark jobs. The recommended approach is to use client mode which means that the spark driver log is retained, inline with the skein job log and is therefore available in the Airflow UI. This was updated and now we are monitoring for stability.
Comment Actions
Change #1111230 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/deployment-charts@master] airflow: Allow the scheduler to patch existing pods
Comment Actions
Change #1111230 merged by jenkins-bot:
[operations/deployment-charts@master] airflow: Allow the scheduler to patch existing pods
Comment Actions
Change #1111237 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/deployment-charts@master] airflow: revert the change to the kube-api networkpolicy
Comment Actions
Change #1111237 merged by jenkins-bot:
[operations/deployment-charts@master] airflow: revert the change to the kube-api networkpolicy
Comment Actions
The airflow-search scheduler is fully migrated to Kubernetes. Per IRC conversation with @dcausse , all outstanding issues have been resolved. As such, I'm closing out this ticket. Feel free to re-open if we missed anything.