Maniphest T390945

Run an experimental dump of 200 regular sized wikis
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	brouberol
	Apr 3 2025, 10:24 AM

Tags

Referenced Files

	F62293020: image.png
	Jun 11 2025, 1:58 PM

Subscribers

Description

Now that we managed to organize our dumpsv1 DAG in a way that we feel comfortable with (cf T390852), we can now try to orchestrate the dumps of a larger number of wikis. We settled on the arbitrary number of 200.

NOTE: we will avoid mediawikiwiki, as it is currently failing (https://phabricator.wikimedia.org/T390839#10705423)

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	airflow: increase pool metrics computation frequency	operations/deployment-charts	master	+4 -1
	airflow: scrape additional metrics	operations/deployment-charts	master	+253 -105
	airflow: set saner performance-related configs	operations/deployment-charts	master	+12 -6

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T88728 Improve Wikimedia dumping infrastructure
		Resolved		BTullis	T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes
		Resolved		brouberol	T388378 Orchestrate dumps v1 from an airflow instance
		Resolved		brouberol	T390945 Run an experimental dump of 200 regular sized wikis
		Resolved		brouberol	T391332 Create a dashboard displaying airflow pool, scheduler, executor and operator activity
		Resolved		brouberol	T391481 Disable XCOMs for dump tasks
		Resolved		brouberol	T391480 Experiment with absolute priority weights
		Resolved		brouberol	T391466 Add more failure tolerance to the dumps DAGs
		Resolved		brouberol	T391497 Adjust the scheduler liveness checks to account for periods of high load
		Resolved		brouberol	T391510 Tolerate dump task pods being OOM killed
		Resolved		brouberol	T391556 Adjust the scheduler resource requirements
		Resolved		brouberol	T391483 Experiment with disabling dynamic task mapping
		Resolved		brouberol	T391678 Adjust the file processing interval
		Resolved		BTullis	T391564 Adjust the database resource requirements
		Resolved		brouberol	T391669 Investigate how to run more than 16 parallel dump task instances
		Resolved		brouberol	T391684 Experiment with running multiple concurrent dump DAGs to keep each graph relatively low in size
		Resolved		brouberol	T391744 Increase file processing timeout
		Resolved		brouberol	T391745 Write a DAG checking that the statically defined list of DAGs matches what NOC returns
		Resolved		BTullis	T391791 Adjust database retention settings
		Resolved		BTullis	T391837 Investigate running the largest wikis with a higher priority weight
		Resolved		brouberol	T392461 Batch dump jobs within a single airflow task to speed up DAG execution
		Resolved		brouberol	T392470 Increase mediawiki-dumps-legacy namespace quotas to enable running 32 parallel pods
		Resolved		brouberol	T392987 Adjust webserver resources
		Resolved		brouberol	T392995 Completed dump pods take a long time to get terminated

Event Timeline

brouberol created this task.Apr 3 2025, 10:24 AM

brouberol triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 3 2025, 10:24 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1212

test_k8s/dumpsv1: introduce a way to exclude certain wikis from a dag run

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1214

Increase the max active tasks from 6 to 16, to speed up the DAG execution

bking subscribed.Apr 3 2025, 1:48 PM

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.Apr 4 2025, 8:29 AM

Gehel edited projects, added Data-Platform-SRE (2025.03.22 - 2025.04.11); removed Data-Platform-SRE.

BTullis added a subtask: T391087: Investigate an incident where the airflow-test-k8s database was wiped.Apr 7 2025, 1:21 PM

Change #1134985 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: set saner performance-related configs

https://gerrit.wikimedia.org/r/1134985

gerritbot added a project: Patch-For-Review.Apr 8 2025, 8:46 AM

brouberol added a subtask: T391332: Create a dashboard displaying airflow pool, scheduler, executor and operator activity.Apr 8 2025, 9:31 AM

Change #1134985 merged by Brouberol:

[operations/deployment-charts@master] airflow: set saner performance-related configs

https://gerrit.wikimedia.org/r/1134985

Change #1135001 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: scrape additional metrics

https://gerrit.wikimedia.org/r/1135001

Change #1135419 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: increase pool metrics computation frequency

https://gerrit.wikimedia.org/r/1135419

brouberol changed the status of subtask T391332: Create a dashboard displaying airflow pool, scheduler, executor and operator activity from Open to In Progress.Apr 9 2025, 1:40 PM

brouberol added a subtask: T391481: Disable XCOMs for dump tasks.Apr 9 2025, 3:24 PM

brouberol added a subtask: T391480: Experiment with absolute priority weights.

brouberol added a subtask: T391466: Add more failure tolerance to the dumps DAGs.

brouberol closed subtask T391466: Add more failure tolerance to the dumps DAGs as Resolved.Apr 9 2025, 3:58 PM

brouberol closed subtask T391481: Disable XCOMs for dump tasks as Resolved.Apr 9 2025, 4:01 PM

brouberol added a subtask: T391497: Adjust the scheduler liveness checks to account for periods of high load.Apr 9 2025, 4:19 PM

brouberol added a subtask: T391510: Tolerate dump task pods being OOM killed.Apr 9 2025, 6:13 PM

brouberol added a subtask: T391556: Adjust the scheduler resource requirements.Apr 10 2025, 7:41 AM

brouberol added a subtask: T391483: Experiment with disabling dynamic task mapping.Apr 10 2025, 7:48 AM

brouberol added a subtask: T391564: Adjust the database resource requirements.Apr 10 2025, 9:18 AM

Change #1135001 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: scrape additional metrics

https://gerrit.wikimedia.org/r/1135001

Change #1135419 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: increase pool metrics computation frequency

https://gerrit.wikimedia.org/r/1135419

brouberol closed subtask T391510: Tolerate dump task pods being OOM killed as Resolved.Apr 10 2025, 9:37 AM

brouberol closed subtask T391332: Create a dashboard displaying airflow pool, scheduler, executor and operator activity as Resolved.

BTullis closed subtask T391087: Investigate an incident where the airflow-test-k8s database was wiped as Resolved.Apr 10 2025, 11:48 AM

brouberol removed a subtask: T391087: Investigate an incident where the airflow-test-k8s database was wiped.Apr 10 2025, 1:28 PM

brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 10 2025, 3:01 PM

brouberol changed the status of subtask T391480: Experiment with absolute priority weights from Open to In Progress.Apr 10 2025, 7:03 PM

brouberol closed subtask T391480: Experiment with absolute priority weights as Resolved.Apr 11 2025, 7:33 AM

brouberol closed subtask T391556: Adjust the scheduler resource requirements as Resolved.

brouberol changed the status of subtask T391483: Experiment with disabling dynamic task mapping from Open to In Progress.Apr 11 2025, 10:10 AM

brouberol added a subtask: T391669: Investigate how to run more than 16 parallel dump task instances.Apr 11 2025, 10:15 AM

Gehel edited projects, added Data-Platform-SRE (2025.04.12 - 2025.05.02); removed Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:14 PM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.

brouberol added a subtask: T391684: Experiment with running multiple concurrent dump DAGs to keep each graph relatively low in size.Apr 11 2025, 1:27 PM

brouberol added a subtask: T391726: Experiment with running multiple scheduler instances.Apr 11 2025, 8:38 PM

brouberol reopened subtask T391510: Tolerate dump task pods being OOM killed as In Progress.Apr 12 2025, 6:44 AM

brouberol added a subtask: T391744: Increase file processing timeout.Apr 12 2025, 1:12 PM

brouberol added a subtask: T391745: Write a DAG checking that the statically defined list of DAGs matches what NOC returns.Apr 12 2025, 1:16 PM

brouberol closed subtask T391483: Experiment with disabling dynamic task mapping as Resolved.

brouberol changed the status of subtask T391497: Adjust the scheduler liveness checks to account for periods of high load from Open to In Progress.Apr 14 2025, 9:43 AM

brouberol changed the status of subtask T391744: Increase file processing timeout from Open to In Progress.Apr 14 2025, 11:25 AM

brouberol added a subtask: T391791: Adjust database retention settings.Apr 14 2025, 2:14 PM

brouberol added a subtask: T391837: Investigate running the largest wikis with a higher priority weight.Apr 14 2025, 2:15 PM

brouberol closed subtask T391744: Increase file processing timeout as Resolved.Apr 15 2025, 7:39 AM

brouberol closed subtask T391497: Adjust the scheduler liveness checks to account for periods of high load as Resolved.Apr 15 2025, 8:05 AM

brouberol changed the status of subtask T391669: Investigate how to run more than 16 parallel dump task instances from Open to In Progress.Apr 15 2025, 9:37 AM

brouberol changed the status of subtask T391745: Write a DAG checking that the statically defined list of DAGs matches what NOC returns from Open to In Progress.Apr 18 2025, 7:20 AM

brouberol closed subtask T391669: Investigate how to run more than 16 parallel dump task instances as Resolved.Apr 18 2025, 8:53 AM

brouberol reopened subtask T391669: Investigate how to run more than 16 parallel dump task instances as In Progress.

brouberol closed subtask T391745: Write a DAG checking that the statically defined list of DAGs matches what NOC returns as Resolved.

brouberol closed subtask T391510: Tolerate dump task pods being OOM killed as Resolved.Apr 23 2025, 8:41 AM

brouberol added a subtask: T392461: Batch dump jobs within a single airflow task to speed up DAG execution.Apr 23 2025, 8:44 AM

brouberol added a subtask: T392470: Increase mediawiki-dumps-legacy namespace quotas to enable running 32 parallel pods.Apr 23 2025, 11:12 AM

brouberol changed the status of subtask T392470: Increase mediawiki-dumps-legacy namespace quotas to enable running 32 parallel pods from Open to In Progress.Apr 23 2025, 11:14 AM

brouberol closed subtask T392470: Increase mediawiki-dumps-legacy namespace quotas to enable running 32 parallel pods as Resolved.Apr 23 2025, 12:36 PM

brouberol reopened subtask T392470: Increase mediawiki-dumps-legacy namespace quotas to enable running 32 parallel pods as In Progress.Apr 23 2025, 12:59 PM

brouberol closed subtask T392470: Increase mediawiki-dumps-legacy namespace quotas to enable running 32 parallel pods as Resolved.Apr 24 2025, 6:52 AM

brouberol closed subtask T391669: Investigate how to run more than 16 parallel dump task instances as Resolved.Apr 24 2025, 12:43 PM

brouberol reopened subtask T391510: Tolerate dump task pods being OOM killed as In Progress.Apr 24 2025, 2:02 PM

brouberol changed the status of subtask T392461: Batch dump jobs within a single airflow task to speed up DAG execution from Open to In Progress.Apr 29 2025, 10:15 AM

brouberol closed subtask T392461: Batch dump jobs within a single airflow task to speed up DAG execution as Resolved.Apr 29 2025, 12:10 PM

brouberol changed the status of subtask T391684: Experiment with running multiple concurrent dump DAGs to keep each graph relatively low in size from Open to In Progress.Apr 29 2025, 2:06 PM

brouberol added a subtask: T392987: Adjust webserver resources.Apr 30 2025, 8:24 AM

brouberol changed the status of subtask T392987: Adjust webserver resources from Open to In Progress.

brouberol added a subtask: T392995: Completed dump pods take a long time to get terminated.Apr 30 2025, 9:22 AM

brouberol changed the status of subtask T392995: Completed dump pods take a long time to get terminated from Open to In Progress.Apr 30 2025, 9:26 AM

brouberol closed subtask T392987: Adjust webserver resources as Resolved.Apr 30 2025, 10:56 AM

brouberol closed subtask T391684: Experiment with running multiple concurrent dump DAGs to keep each graph relatively low in size as Resolved.May 5 2025, 10:10 AM

brouberol closed subtask T392995: Completed dump pods take a long time to get terminated as Resolved.

Gehel edited projects, added Data-Platform-SRE (2025.05.02 - 2025.05.23); removed Data-Platform-SRE (2025.04.12 - 2025.05.02).May 5 2025, 12:49 PM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.

Gehel edited projects, added Data-Platform-SRE (2025.05.24 - 2025.06.13); removed Data-Platform-SRE (2025.05.02 - 2025.05.23).May 23 2025, 1:08 PM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.

brouberol closed subtask T391510: Tolerate dump task pods being OOM killed as Resolved.Jun 10 2025, 3:26 PM

I think that we can call this done.

image.png (728×1 px, 120 KB)

We have now split our DAGs so that we have around ~145 wikis per DAG and I had two runs of different groups complete successfully yesterday.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.Jun 13 2025, 8:16 AM

BTullis closed subtask T391837: Investigate running the largest wikis with a higher priority weight as Resolved.Jun 13 2025, 3:56 PM

BTullis closed subtask T391564: Adjust the database resource requirements as Resolved.Jun 24 2025, 3:08 PM

BTullis closed subtask T391791: Adjust database retention settings as Resolved.Jun 27 2025, 1:42 PM

BTullis removed a subtask: T391726: Experiment with running multiple scheduler instances.Jul 2 2025, 12:35 PM