Maniphest T398436

Future improvements to the Dumps on Airflow configuration - tracking task
Open, Needs TriagePublic
Actions

Assigned To

Authored By

	BTullis
	Jul 2 2025, 12:19 PM

Tags

Referenced Files

None

Subscribers

Description

During the work on T352650: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes we have identified a number of areas for potential improvement that remain outstanding.
This ticket is a way for us to collate and track these potential improvements, post the switch over.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Update the container used for the dumps toolbox	operations/deployment-charts	master	+1 -1

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	test_k8s/dumps/sql-xml: enable skipping a single stage for a given wiki	repos/data-engineering/airflow-dags!1552	brouberol	T398436	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		brouberol	T398436 Future improvements to the Dumps on Airflow configuration - tracking task
		Open		None	T389771 Display a link to the dump HTML page in the airflow dump task instance page
		Open		None	T389774 Automatically create a phabricator ticket when a dump SLA is missed
		Open		None	T389783 Create more sophisticated monitors for the various dump types
		Resolved		brouberol	T389785 Adjust the pool parallelism settings after having performed a load test of the dumpsv1 process
		Open		None	T389927 Display a link to the Grafana pod usage dashboard in the airflow dump task instance page
		Open		None	T391726 Experiment with running multiple scheduler instances
		Resolved		brouberol	T398609 Reduce memory requirements and startup time for dumps v1 sql/xml dumps
		Resolved		brouberol	T398649 Allow the mediawiki pagetitles amd imageinfo dumps DAGs to backfill
		Resolved		brouberol	T398766 Create the xml/sql dumps directory per wiki before starting the actual dumps
		Resolved		brouberol	T398772 Adjust cirrussearch dump pod memory requirement per DB section
		Resolved		brouberol	T398788 Imageinfo dumps are failing
		Resolved		brouberol	T398797 SQL/XML dump for enwiki is much slower than anticipated
		In Progress		brouberol	T398800 Dumps on Airflow are not using the dbstore servers due to etcd timeout
		Resolved		brouberol	T398911 The shorturl dumps jub is misconfigured
		Resolved		brouberol	T399176 We should skip the metacurrentdumprecombine stage for mediawikiwiki

Event Timeline

BTullis created this task.Jul 2 2025, 12:19 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 2 2025, 12:19 PM

BTullis moved this task from Incoming to Incidents / Follow up on the Data-Platform-SRE board.Jul 2 2025, 12:19 PM

BTullis added a subtask: T389771: Display a link to the dump HTML page in the airflow dump task instance page.Jul 2 2025, 12:26 PM

BTullis added a subtask: T389774: Automatically create a phabricator ticket when a dump SLA is missed.

BTullis added a subtask: T389783: Create more sophisticated monitors for the various dump types.Jul 2 2025, 12:32 PM

BTullis added a subtask: T389785: Adjust the pool parallelism settings after having performed a load test of the dumpsv1 process.

BTullis added a subtask: T389927: Display a link to the Grafana pod usage dashboard in the airflow dump task instance page.

BTullis added a subtask: T391726: Experiment with running multiple scheduler instances.Jul 2 2025, 12:35 PM

BTullis mentioned this in T352650: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes.Jul 2 2025, 1:12 PM

brouberol added a subtask: T398609: Reduce memory requirements and startup time for dumps v1 sql/xml dumps.Jul 3 2025, 10:32 AM

brouberol changed the status of subtask T398609: Reduce memory requirements and startup time for dumps v1 sql/xml dumps from Open to In Progress.

brouberol added a subtask: T398649: Allow the mediawiki pagetitles amd imageinfo dumps DAGs to backfill.Jul 3 2025, 4:50 PM

brouberol changed the status of subtask T398649: Allow the mediawiki pagetitles amd imageinfo dumps DAGs to backfill from Open to In Progress.Jul 3 2025, 4:57 PM

brouberol closed subtask T398649: Allow the mediawiki pagetitles amd imageinfo dumps DAGs to backfill as Resolved.Jul 4 2025, 10:23 AM

brouberol closed subtask T389785: Adjust the pool parallelism settings after having performed a load test of the dumpsv1 process as Resolved.Jul 4 2025, 3:28 PM

brouberol added a subtask: T398766: Create the xml/sql dumps directory per wiki before starting the actual dumps.Jul 6 2025, 8:39 AM

brouberol changed the status of subtask T398766: Create the xml/sql dumps directory per wiki before starting the actual dumps from Open to In Progress.Jul 6 2025, 8:46 AM

brouberol added a subtask: T398772: Adjust cirrussearch dump pod memory requirement per DB section.Jul 6 2025, 4:50 PM

brouberol added a subtask: T398788: Imageinfo dumps are failing.Jul 7 2025, 6:45 AM

brouberol changed the status of subtask T398772: Adjust cirrussearch dump pod memory requirement per DB section from Open to In Progress.Jul 7 2025, 6:47 AM

brouberol closed subtask T398788: Imageinfo dumps are failing as Resolved.Jul 7 2025, 7:45 AM

brouberol added a subtask: T398797: SQL/XML dump for enwiki is much slower than anticipated.Jul 7 2025, 8:41 AM

brouberol changed the status of subtask T398797: SQL/XML dump for enwiki is much slower than anticipated from Open to In Progress.

BTullis added a subtask: T398800: Dumps on Airflow are not using the dbstore servers due to etcd timeout.Jul 7 2025, 8:56 AM

brouberol closed subtask T398766: Create the xml/sql dumps directory per wiki before starting the actual dumps as Resolved.Jul 7 2025, 8:58 AM

brouberol changed the status of subtask T398800: Dumps on Airflow are not using the dbstore servers due to etcd timeout from Open to In Progress.Jul 7 2025, 9:21 AM

brouberol closed subtask T398772: Adjust cirrussearch dump pod memory requirement per DB section as Resolved.Jul 7 2025, 11:55 AM

brouberol added a subtask: T398911: The shorturl dumps jub is misconfigured.Jul 8 2025, 7:27 AM

brouberol changed the status of subtask T398911: The shorturl dumps jub is misconfigured from Open to In Progress.Jul 8 2025, 7:32 AM

brouberol closed subtask T398911: The shorturl dumps jub is misconfigured as Resolved.Jul 8 2025, 12:26 PM

brouberol claimed this task.Jul 10 2025, 10:55 AM

brouberol added a subtask: T399176: We should skip the metacurrentdumprecombine stage for mediawikiwiki.

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1552

test_k8s/dumps/sql-xml: enable skipping a single stage for a given wiki

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1552

test_k8s/dumps/sql-xml: enable skipping a single stage for a given wiki

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2025, 11:30 AM

BTullis closed subtask T399176: We should skip the metacurrentdumprecombine stage for mediawikiwiki as Resolved.Jul 14 2025, 9:39 AM

brouberol closed subtask T398609: Reduce memory requirements and startup time for dumps v1 sql/xml dumps as Resolved.Aug 5 2025, 12:32 PM

brouberol closed subtask T398797: SQL/XML dump for enwiki is much slower than anticipated as Resolved.Aug 5 2025, 1:46 PM

Change #1265478 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the container used for the dumps toolbox

https://gerrit.wikimedia.org/r/1265478

gerritbot added a project: Patch-For-Review.Tue, Mar 31, 4:19 PM

Change #1265478 merged by jenkins-bot:

[operations/deployment-charts@master] Update the container used for the dumps toolbox

https://gerrit.wikimedia.org/r/1265478

Maintenance_bot removed a project: Patch-For-Review.Tue, Mar 31, 5:31 PM