Maniphest T389962

Differentiate full and partial dumps in the DAGs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	brouberol
	Mar 25 2025, 3:34 PM

Tags

Referenced Files

	F62313518: image.png
	Jun 13 2025, 2:19 PM

Subscribers

Description

The dumps v1 currently run in 2 main phases:

full dumps, scheduled one the first day of the month
partial dumps, scheduled on the 20th day of the month

We currently don't differentiate (nor schedule) these runs, so we need to ensure that we mirror the current setup in our airflow config.

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Dumps_v1: Dirrerentiate between full and partial dumps	repos/data-engineering/airflow-dags!1402	btullis	dumps_full_partial	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T88728 Improve Wikimedia dumping infrastructure
		Resolved		BTullis	T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes
		Resolved		brouberol	T388378 Orchestrate dumps v1 from an airflow instance
		Resolved		BTullis	T389962 Differentiate full and partial dumps in the DAGs

Event Timeline

brouberol created this task.Mar 25 2025, 3:34 PM

brouberol triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 25 2025, 3:34 PM

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.Mar 26 2025, 9:47 AM

Gehel moved this task from Scratch to Quarterly Goals on the Data-Platform-SRE board.Mar 26 2025, 9:50 AM

BTullis claimed this task.Jun 5 2025, 6:28 PM

BTullis edited projects, added Data-Platform-SRE (2025.05.24 - 2025.06.13); removed Data-Platform-SRE.

BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.

Gehel edited projects, added Data-Platform-SRE (2025.06.13 - 2025.07.04); removed Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:49 AM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1402

Dumps_v1: Dirrerentiate between full and partial dumps

This is now done.

image.png (690×1 px, 148 KB)

The schedules are now applied, so we will be doing full sized dumps on the correct schedule.

BTullis closed this task as Resolved.Jun 13 2025, 2:22 PM

BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.Jun 20 2025, 8:54 AM