Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current
Open, In Progress, Needs TriagePublic
Actions

Assigned To

Authored By

	xcollazo
	Jun 4 2025, 1:58 PM

Description

Since making the new MW Content Tables production quality, we have marked the old snapshot based tables deprecated:

table	deprecation date
`wmf.mediawiki_wikitext_history`	2025-01-29
`wmf.mediawiki_wikitext_current`	2025-04-30

A cursory search shows that the history table is only being referenced by our own DPE code.

For the current table, our own code, plus one deprecated project references the table.

In this task we should:

Figure out the ownership of generate_anchor_dictionary_spark.py code, and get a commitment to migrate code.
- Code is indeed being migrated via T398950. See reference to new tables here.
Remove code that generates the content of both tables
- remove code that imports from dumps servers to HDFS at modules/profile/manifests/analytics/refinery/job/import_mediawiki_dumps.pp
- remove data purge code at modules/profile/manifests/analytics/refinery/job/data_purge.pp
- remove airflow jobs
Remove tables
Remove any remaining data from the raw imports at /mnt/hdfs/wmf/data/raw/mediawiki/dumps

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	analytics: Remove rsync scripts that import Dumps 1 XML into HDFS	operations/puppet	production	+3 -56
	analytics: Absent rsync scripts that import Dumps 1 XML into HDFS	operations/puppet	production	+5 -5

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	analytics: remove mediawiki_wikitext_* DAGs	repos/data-engineering/airflow-dags!1578	xcollazo	remove-wikitext-jobs	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Open	None	T366752 Dumps 2.0 Phase III: Production level dumps (SDS 1.2)
In Progress	xcollazo	T384382 Production-level file export (aka dump) of MW Content in XML
In Progress	xcollazo	T396031 Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current

Event Timeline

xcollazo created this task.Jun 4 2025, 1:58 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2025, 1:58 PM

Ahoelzl edited projects, added Data-Engineering (Q4 2025 April 1st - June 30th); removed Data-Engineering.Jun 4 2025, 2:25 PM

xcollazo added a parent task: T366752: Dumps 2.0 Phase III: Production level dumps (SDS 1.2).Jun 4 2025, 2:46 PM

mfossati subscribed.Jun 13 2025, 9:39 AM

xcollazo updated the task description. (Show Details)Jul 8 2025, 1:47 PM

Change #1167224 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] analytics: Absent rsync scripts that import Dumps 1 XML into HDFS

https://gerrit.wikimedia.org/r/1167224

gerritbot added a project: Patch-For-Review.Jul 8 2025, 1:55 PM

xcollazo changed the task status from Open to In Progress.Jul 8 2025, 3:54 PM

xcollazo claimed this task.

Change #1167224 merged by Btullis:

[operations/puppet@production] analytics: Absent rsync scripts that import Dumps 1 XML into HDFS

https://gerrit.wikimedia.org/r/1167224

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2025, 3:30 PM

xcollazo moved this task from Next Up to In progress on the Data-Engineering (Q4 2025 April 1st - June 30th) board.Jul 10 2025, 4:43 PM

xcollazo updated the task description. (Show Details)Jul 22 2025, 2:18 PM

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1578

analytics: remove mediawiki_wikitext_* DAGs

dr0ptp4kt subscribed.Jul 23 2025, 8:42 PM

Change #1172113 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] analytics: Remove rsync scripts that import Dumps 1 XML into HDFS

https://gerrit.wikimedia.org/r/1172113

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1578

analytics: remove mediawiki_wikitext_* DAGs

xcollazo updated the task description. (Show Details)Jul 24 2025, 1:29 PM

xcollazo updated the task description. (Show Details)

Manually deleted the db representation for the following DAGs from the Airflow UI:

mediawiki_wikitext_current
mediawiki_wikitext_history
wikidata_wikitext_history

Change #1172113 merged by Btullis:

[operations/puppet@production] analytics: Remove rsync scripts that import Dumps 1 XML into HDFS

https://gerrit.wikimedia.org/r/1172113

Maintenance_bot removed a project: Patch-For-Review.Jul 25 2025, 12:30 PM

xcollazo updated the task description. (Show Details)Jul 25 2025, 4:25 PM

Removed remaining Dumps1 XML artifacts from HDFS:

$ hostname -f
an-launcher1002.eqiad.wmnet
$ whoami 
analytics

$ kerberos-run-command analytics hdfs dfs -ls /wmf/data/raw/mediawiki/dumps
Found 3 items
drwxr-x---   - analytics analytics-privatedata-users          0 2025-07-01 05:00 /wmf/data/raw/mediawiki/dumps/pages_meta_current
drwxr-x---   - analytics analytics-privatedata-users          0 2025-07-01 03:00 /wmf/data/raw/mediawiki/dumps/pages_meta_history
drwxr-x---   - analytics analytics-privatedata-users          0 2025-07-20 05:00 /wmf/data/raw/mediawiki/dumps/siteinfo_namespaces

$ kerberos-run-command analytics hdfs dfs -du -h /wmf/data/raw/mediawiki/dumps/pages_meta*
406.0 G  /wmf/data/raw/mediawiki/dumps/pages_meta_current/20250401
408.4 G  /wmf/data/raw/mediawiki/dumps/pages_meta_current/20250501
411.5 G  /wmf/data/raw/mediawiki/dumps/pages_meta_current/20250601
406.2 G  /wmf/data/raw/mediawiki/dumps/pages_meta_current/20250701
6.3 T  /wmf/data/raw/mediawiki/dumps/pages_meta_history/20250401
6.4 T  /wmf/data/raw/mediawiki/dumps/pages_meta_history/20250501
6.4 T  /wmf/data/raw/mediawiki/dumps/pages_meta_history/20250601
4.0 T  /wmf/data/raw/mediawiki/dumps/pages_meta_history/20250701

$ kerberos-run-command analytics hdfs dfs -rm -r /wmf/data/raw/mediawiki/dumps/pages_meta*
25/07/25 16:33:57 INFO fs.TrashPolicyDefault: Moved: 'hdfs://analytics-hadoop/wmf/data/raw/mediawiki/dumps/pages_meta_current' to trash at: hdfs://analytics-hadoop/user/analytics/.Trash/Current/wmf/data/raw/mediawiki/dumps/pages_meta_current
25/07/25 16:33:57 INFO fs.TrashPolicyDefault: Moved: 'hdfs://analytics-hadoop/wmf/data/raw/mediawiki/dumps/pages_meta_history' to trash at: hdfs://analytics-hadoop/user/analytics/.Trash/Current/wmf/data/raw/mediawiki/dumps/pages_meta_history

Mentioned in SAL (#wikimedia-analytics) [2025-07-25T16:36:40Z] <xcollazo> removed remaining raw Dumps1 XML files from HDFS. See T396031#11035363 for details.

xcollazo removed a parent task: T366752: Dumps 2.0 Phase III: Production level dumps (SDS 1.2).Jul 25 2025, 5:55 PM

xcollazo added a parent task: T384382: Production-level file export (aka dump) of MW Content in XML.

The only remaining task here is to DROP the tables themselves. Will do that soon.

Ahoelzl edited projects, added Data-Engineering (Q1 FY25/26 July 1st - September 30th); removed Data-Engineering (Q4 2025 April 1st - June 30th).Jul 25 2025, 10:56 PM

Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.

xcollazo moved this task from In progress to Blocked/Paused on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Aug 12 2025, 5:26 PM

Ahoelzl edited projects, added Data-Engineering (Q2 FY25/26 October 1st - December 31th); removed Data-Engineering (Q1 FY25/26 July 1st - September 30th).Oct 9 2025, 12:21 AM

Ahoelzl moved this task from Next Up to Blocked/Paused on the Data-Engineering (Q2 FY25/26 October 1st - December 31th) board.