⚓ T418442 Finish and deploy scraper Airflow job

Title	Reference	Author	Source Branch	Dest Branch
Only keep the latest month of Cite per-page data	repos/wmde/analytics!26	awight	delete-more	main
Increase Cite Refs scraper memory to 6GB	repos/data-engineering/airflow-dags!2071	awight	more-mem	main
Rely on SkeinOperator krb5 setup	repos/data-engineering/airflow-dags!2070	awight	krb5-defaults	main

Status	Assigned	Task
Open	None	T396752 [Epic] Metrics for sub-referencing
Open	None	T398277 [Epic] Scraping data dumps and scraper enhancements
Open	None	T396731 Scraper: Build a new dashboard for the updated scraper data
Open	None	T412019 [Epic] Schedule scraper and aggregations as an Airflow job
Resolved	None	T418442 Finish and deploy scraper Airflow job
Invalid	awight	T419654 Append environment for BashSensor
Open	None	T419655 Migrate Tech Wishes scraper to gitlab.wikimedia.org

awight created this task.Feb 26 2026, 6:43 AM

awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2026-02-17-Beautiful-Beetroots board.

awight updated the task description. (Show Details)Feb 26 2026, 8:25 AM

awight updated the task description. (Show Details)Feb 26 2026, 8:28 AM

awight updated the task description. (Show Details)Feb 26 2026, 12:57 PM

awight updated the task description. (Show Details)

awight updated the task description. (Show Details)Feb 26 2026, 3:23 PM

awight updated the task description. (Show Details)Feb 26 2026, 4:21 PM

(CC'ing data platform engineers who have generously helped us, and who might be interested in watching the exciting conclusion.)

awight updated the task description. (Show Details)Feb 27 2026, 11:33 AM

awight updated the task description. (Show Details)

Tobi_WMDE_SW edited projects, added WMDE-TechWish-Sprint-2026-03-03-Spinach; removed WMDE-TechWish-Sprint-2026-02-17-Beautiful-Beetroots.Mar 3 2026, 9:06 AM

Tobi_WMDE_SW moved this task from Sprint Backlog to Tech Review on the WMDE-TechWish-Sprint-2026-03-03-Spinach board.Mar 3 2026, 9:06 AM

Tobi_WMDE_SW added a project: WMDE-TechWish-Sprint-2026-02-17-Beautiful-Beetroots.Mar 3 2026, 9:08 AM

awight updated the task description. (Show Details)Mar 4 2026, 1:26 PM

awight opened https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/23

Template all source and destination table names in Cite refs script

thiemowmde updated the task description. (Show Details)Mar 9 2026, 10:13 AM

andrewtavis-wmde merged https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/23

Template all source and destination table names in Cite refs script

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2026, 12:30 PM

awight updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1957

New job: scrape and aggregate page summaries

awight opened https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/24

Parameterize another table name

andrewtavis-wmde merged https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/24

Parameterize another table name

awight opened https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/25

Yet another param that was missed

andrewtavis-wmde merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1957

New job: scrape and aggregate page summaries

awight updated the task description. (Show Details)Mar 10 2026, 4:50 PM

I found that the BashSensor is not getting the correct KRB5CCNAME, and I'll have to implement an append_env which adds my additional variables to the executor's environment.

What I find is that these variables can be extracted from the DAG parse os.environ:

HADOOP_CONF_DIR: /etc/hadoop/conf
KRB5CCNAME: /tmp/airflow_krb5_ccache/krb5cc
KRB5_CONFIG: /etc/krb5.conf
KRB5_KEYTAB: /etc/kerberos/keytabs/airflow.keytab
KRB5_PRINCIPAL: analytics-wmde/airflow-wmde.discovery.wmnet

However, the k8s pod spec places the krb5cc somewhere else:

mountPath: "/tmp/airflow_krb5_ccache"

This causes my sensor to fail on its first attempt to access Hadoop:

kinit: Failed to store credentials: No credentials cache found (filename: /tmp/airflow_krb5_ccache/krb5cc) while getting initial credentials

awight created subtask T419654: Append environment for BashSensor.Mar 11 2026, 7:27 AM

awight created subtask T419655: Migrate Tech Wishes scraper to gitlab.wikimedia.org.Mar 11 2026, 7:56 AM

awight closed subtask T419654: Append environment for BashSensor as Invalid.Mar 11 2026, 9:13 AM

awight updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2070

Rely on SkeinOperator krb5 setup

andrewtavis-wmde merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2070

Rely on SkeinOperator krb5 setup

The job seems to be hitting an out-of-memory error now. I increased the memory from 2GB to 4GB but now it crashes in the third chunk of dewiki.

This is surprising because earlier runs showed flat memory usage, well under 2GB.

Running locally shows that memory stabilizes just above 4GB, so I'll set the job limit to 6GB. We'll need to come back later and diagnose whether there is a new memory leak or duplication, maybe caused by the change in output methods.