Page MenuHomePhabricator
Create Task
Maniphest T362870

Define a strategy to deal with xml-dumps huge files on the datalake
Open, Needs TriagePublic
Actions

Description

Some files are of whole-history xml-dumps are very big (larger than 10Gb). In particular, one file of enwiki weights more than 65Gb :(
We should strive to find a solution on how to process them efficiently, cause for now they are processed through a single worker due to the hierarchical structure of the dumps, and this is very inefficient and subject to failures.

joal@stat1008:/mnt/hdfs/wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401$ find . -type f -size +10G
./cawiki/cawiki-20240401-pages-meta-history.xml.bz2
./cswiki/cswiki-20240401-pages-meta-history.xml.bz2
./dewiki/dewiki-20240401-pages-meta-history5.xml-p6991011p7076405.bz2
./dewiki/dewiki-20240401-pages-meta-history5.xml-p9023811p9082351.bz2
./elwiki/elwiki-20240401-pages-meta-history.xml.bz2
./enwiki/enwiki-20240401-pages-meta-history10.xml-p5124527p5137513.bz2
./enwiki/enwiki-20240401-pages-meta-history21.xml-p36250030p36395491.bz2
./enwikiquote/enwikiquote-20240401-pages-meta-history.xml.bz2
./enwikivoyage/enwikivoyage-20240401-pages-meta-history.xml.bz2
./enwiktionary/enwiktionary-20240401-pages-meta-history.xml.bz2
./fawiki/fawiki-20240401-pages-meta-history.xml.bz2
./fiwiki/fiwiki-20240401-pages-meta-history.xml.bz2
./frwiki/frwiki-20240401-pages-meta-history5.xml-p7676440p7846557.bz2
./frwikisource/frwikisource-20240401-pages-meta-history.xml.bz2
./nowiki/nowiki-20240401-pages-meta-history.xml.bz2
./ruwiki/ruwiki-20240401-pages-meta-history1.xml-p139579p148255.bz2
./ruwikisource/ruwikisource-20240401-pages-meta-history.xml.bz2
./srwiki/srwiki-20240401-pages-meta-history.xml.bz2
./trwiki/trwiki-20240401-pages-meta-history.xml.bz2



joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/cawiki/cawiki-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      18.4 G 2024-04-05 08:26 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/cawiki/cawiki-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/cswiki/cswiki-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      13.2 G 2024-04-04 03:39 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/cswiki/cswiki-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/dewiki/dewiki-20240401-pages-meta-history5.xml-p6991011p7076405.bz2
-rw-r-----   3 analytics analytics-privatedata-users      10.5 G 2024-04-05 03:03 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/dewiki/dewiki-20240401-pages-meta-history5.xml-p6991011p7076405.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/dewiki/dewiki-20240401-pages-meta-history5.xml-p9023811p9082351.bz2
-rw-r-----   3 analytics analytics-privatedata-users      13.0 G 2024-04-05 03:12 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/dewiki/dewiki-20240401-pages-meta-history5.xml-p9023811p9082351.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/elwiki/elwiki-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      11.4 G 2024-04-04 04:01 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/elwiki/elwiki-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwiki/enwiki-20240401-pages-meta-history10.xml-p5124527p5137513.bz2
-rw-r-----   3 analytics analytics-privatedata-users      66.4 G 2024-04-07 05:59 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwiki/enwiki-20240401-pages-meta-history10.xml-p5124527p5137513.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwiki/enwiki-20240401-pages-meta-history21.xml-p36250030p36395491.bz2
-rw-r-----   3 analytics analytics-privatedata-users      16.8 G 2024-04-07 06:23 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwiki/enwiki-20240401-pages-meta-history21.xml-p36250030p36395491.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwikiquote/enwikiquote-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      12.3 G 2024-04-04 04:30 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwikiquote/enwikiquote-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwikivoyage/enwikivoyage-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      10.3 G 2024-04-04 04:20 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwikivoyage/enwikivoyage-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwiktionary/enwiktionary-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      40.2 G 2024-04-05 07:42 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/enwiktionary/enwiktionary-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/fawiki/fawiki-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      21.0 G 2024-04-05 08:22 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/fawiki/fawiki-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/fiwiki/fiwiki-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      13.5 G 2024-04-04 03:16 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/fiwiki/fiwiki-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/frwiki/frwiki-20240401-pages-meta-history5.xml-p7676440p7846557.bz2
-rw-r-----   3 analytics analytics-privatedata-users      12.4 G 2024-04-07 09:20 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/frwiki/frwiki-20240401-pages-meta-history5.xml-p7676440p7846557.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/frwikisource/frwikisource-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      11.9 G 2024-04-04 03:57 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/frwikisource/frwikisource-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/nowiki/nowiki-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      13.6 G 2024-04-04 03:06 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/nowiki/nowiki-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/ruwiki/ruwiki-20240401-pages-meta-history1.xml-p139579p148255.bz2
-rw-r-----   3 analytics analytics-privatedata-users      10.7 G 2024-04-06 03:17 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/ruwiki/ruwiki-20240401-pages-meta-history1.xml-p139579p148255.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/ruwikisource/ruwikisource-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      13.1 G 2024-04-04 04:26 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/ruwikisource/ruwikisource-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/srwiki/srwiki-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      12.6 G 2024-04-05 08:35 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/srwiki/srwiki-20240401-pages-meta-history.xml.bz2

joal@stat1008:~/enwi8ki_huge_file$ hdfs dfs -ls -h /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/trwiki/trwiki-20240401-pages-meta-history.xml.bz2
-rw-r-----   3 analytics analytics-privatedata-users      13.8 G 2024-04-05 07:58 /wmf/data/raw/mediawiki/dumps/pages_meta_history/20240401/trwiki/trwiki-20240401-pages-meta-history.xml.bz2

Event Timeline

JAllemandou created this task.Apr 18 2024, 11:23 AM
Restricted Application added subscribers: Huji, Aklapper. · View Herald TranscriptApr 18 2024, 11:23 AM
JAllemandou renamed this task from Define a strategy to deal with xml-dumps huge files to Define a strategy to deal with xml-dumps huge files on the datalake.Apr 18 2024, 11:30 AM
JAllemandou added a comment.Apr 18 2024, 12:55 PM

Some of the big files listed above are due to the dumps job not splitting files (for cawiki and cswiki for instance).
For the rest, big files come from big pages (many revisions and big text):

spark.sql("""
     | SELECT
     |   wiki_db,
     |   page_id,
     |   count(1) as revision_count,
     |   sum(revision_text_bytes) as text_weight
     | from wmf.mediawiki_history
     |   where snapshot='2024-03'
     |   and event_entity = 'revision'
     |   and event_type = 'create'
     |   and not revision_is_deleted_by_page_deletion
     | group by wiki_db, page_id
     | order by text_weight DESC
     | limit 20
     | """).show(100, false)
+-----------+--------+--------------+------------+                              
|wiki_db    |page_id |revision_count|text_weight |
+-----------+--------+--------------+------------+
|enwiki     |5137507 |1346638       |463589202714|
|ruwiki     |205407  |327734        |98327067901 |
|frwiki     |7846555 |213413        |95076531961 |
|dewiki     |9082349 |373094        |90919451888 |
|commonswiki|1894972 |750375        |88226610099 |
|enwiki     |5149102 |411056        |76082914673 |
|enwiki     |36395484|400807        |74132557523 |
|ruwiki     |148254  |131666        |72615443749 |
|enwiki     |2535910 |505135        |70774375375 |
|dewiki     |7076401 |218421        |66236030573 |
|enwiki     |11424955|220512        |64807978487 |
|enwiki     |972034  |401277        |61837314055 |
|enwiki     |68479621|74790         |57447551495 |
|enwiki     |1470141 |411101        |55771136049 |
|dewiki     |6529924 |200046        |55555153937 |
|zhwiki     |84599   |577396        |54912852986 |
|enwiki     |34745517|378634        |50069312070 |
|zhwiki     |284591  |165702        |49405881699 |
|ruwiki     |15920   |202182        |49323471028 |
|hewiki     |13822   |216934        |48579444220 |
+-----------+--------+--------------+------------+
lbowmaker moved this task from Incoming (new tickets) to Estimated (To be planned) on the Data-Engineering board.Tue, Apr 23, 2:50 PM
lbowmaker moved this task from Estimated (To be planned) to To be estimated/discussed on the Data-Engineering board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL