Maniphest T391837

Investigate running the largest wikis with a higher priority weight
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	brouberol
	Apr 14 2025, 2:15 PM

Project Tags

Referenced Files

	F62314012: image.png
	Jun 13 2025, 3:30 PM

Subscribers

Description

To ensure the huge wikis (enwiki and wikidata) start first, we could play with priority_weight set to a higher value than the other tasks)

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Dumps_v1: Bump up the priority weight for enwiki and wikidatawiki	repos/data-engineering/airflow-dags!1406	btullis	add_weight_huge_wikis	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T88728 Improve Wikimedia dumping infrastructure
		Resolved		BTullis	T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes
		Resolved		brouberol	T388378 Orchestrate dumps v1 from an airflow instance
		Resolved		brouberol	T390945 Run an experimental dump of 200 regular sized wikis
		Resolved		BTullis	T391837 Investigate running the largest wikis with a higher priority weight

Event Timeline

brouberol created this task.Apr 14 2025, 2:15 PM

brouberol triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 14 2025, 2:15 PM

brouberol moved this task from Incoming to Quarterly Goals on the Data-Platform-SRE board.Apr 17 2025, 1:12 PM

BTullis claimed this task.Jun 13 2025, 2:23 PM

BTullis edited projects, added: Data-Platform-SRE (2025.06.13 - 2025.07.04); removed: Data-Platform-SRE.

BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.

I think that's working. Nice and simple.

image.png (1,027×959 px, 157 KB)

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1406

Dumps_v1: Bump up the priority weight for enwiki and wikidatawiki

BTullis closed this task as Resolved.Jun 13 2025, 3:56 PM

BTullis moved this task from Needs Review to Done on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.Jun 20 2025, 8:54 AM