Page MenuHomePhabricator
Create Task
Maniphest T382084

20241201 wikidatawiki xml dump not progressing
Closed, ResolvedPublic
Actions

Description

We've been seeing email alerts like:

[data engineering alerts] [ops-dumps] PASS: dump-jobs-watcher
Systemd timer ran the following command:

    /bin/bash /usr/local/bin/job_watcher.sh --dumpsbasedir /data/xmldatadumps/public --locksbasedir /data/xmldatadumps/private

Its return value was 0 and emitted the following output:

PROBLEM: wikidatawiki has file wikidatawiki/20241201/wikidatawiki-20241201-pages-meta-history9.xml-p12828996p12875404.bz2.inprog at least 4 hours older than lock
...

This looks similar to the symptom reported in T376926: frwiki trashing for 20241001 dump.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT378559 Data Platform Data Incidents
 ResolvedOttomataT382084 20241201 wikidatawiki xml dump not progressing

Event Timeline

Ottomata created this task.Dec 12 2024, 5:49 PM
Ottomata removed a project: Epic.
Ottomata added subscribers: xcollazo, Milimetric.

@xcollazo recommended we try to kill (and restart?) the job, as described in T376926#10218853.

I'll try that now...

Stashbot added a comment.Dec 12 2024, 5:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-12T17:53:57Z] <ottomata> killing wikidatawiki xml dump process to try to unstick it - T382084

Ottomata added a comment.EditedDec 12 2024, 5:55 PM
17:45:21 [@snapshot1011:/home/otto] $ cd /mnt/dumpsdata/xmldatadumps/private/wikidatawiki
17:45:23 [@snapshot1011:/mnt/dumpsdata/xmldatadumps/private/wikidatawiki] $ cat lock_20241201
snapshot1011.eqiad.wmnet 3510981

17:50:49 [@snapshot1011:/mnt/dumpsdata/xmldatadumps/private/wikidatawiki] $ sudo -u dumpsgen  bash
dumpsgen@snapshot1011:/mnt/dumpsdata/xmldatadumps/private/wikidatawiki$ cd /srv/deployment/dumps/dumps/xmldumps-backup

dumpsgen@snapshot1011:/srv/deployment/dumps/dumps/xmldumps-backup$ python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps:bigwikis --wiki wikidatawiki --dryrun
would kill processes ['418228', '418229', '418230', '418236', '418241', '418242', '418243', '418244', '418245', '418246', '418247', '418259', '418262', '418265', '418701', '418732', '418810', '418823', '418826', '418862', '418901', '418906', '419048', '419094', '419136', '419164', '419862', '422623']

python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps:bigwikis --wiki wikidatawiki

dumpsgen@snapshot1011:/srv/deployment/dumps/dumps/xmldumps-backup$ ls -l /mnt/dumpsdata/xmldatadumps/private/wikidatawiki/lock_20241201
-rwxr-xr-x 1 dumpsgen dumpsgen 32 Dec 12 17:54 /mnt/dumpsdata/xmldatadumps/private/wikidatawiki/lock_20241201

dumpsgen@snapshot1011:/srv/deployment/dumps/dumps/xmldumps-backup$ rm /mnt/dumpsdata/xmldatadumps/private/wikidatawiki/lock_20241201

Done. But hm, maybe we should remove the lockfile before killing?

Ottomata added a comment.Dec 12 2024, 7:32 PM

Discussed with Xabriel in slack. I was worried about the fact that there is no lockfile, so I killed the processes again.

dumpsgen@snapshot1011:/srv/deployment/dumps/dumps/xmldumps-backup$ python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps:bigwikis --wiki wikidatawiki --dryrun
would kill processes ['432994', '432995', '432996', '432997', '432998', '432999', '433000', '433001', '433002', '433003', '433004', '433005', '433006', '433007', '433008', '433009', '433010', '433011', '433012', '433013', '433014', '433015', '433018', '433021', '433023', '433026', '433028', '433204', '433290', '433320', '433428', '433731', '433754', '433757', '433760', '433765', '433827', '433839', '433847', '433864', '433894', '433960', '434259', '434328', '434430', '434549', '434556', '438970', '442852', '443110', '444439', '447923']

dumpsgen@snapshot1011:/srv/deployment/dumps/dumps/xmldumps-backup$ python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps:bigwikis --wiki wikidatawiki
Ahoelzl added a comment.Dec 16 2024, 6:48 PM

Still in progress according to https://dumps.wikimedia.org/backup-index.html.

Ahoelzl edited projects, added: Data-Engineering (Q2 2024 October 1st - December 31th); removed: Data-Engineering.Dec 16 2024, 6:51 PM
Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q2 2024 October 1st - December 31th) board.Dec 18 2024, 1:52 PM
Ahoelzl added a project: Dumps-Generation.
Restricted Application added a project: Data-Platform. · View Herald TranscriptDec 18 2024, 1:52 PM
xcollazo added a comment.Dec 20 2024, 3:38 PM

Looks like last logs from wikidatawiki 2024-12-01 run are from 2024-12-12:

xcollazo@snapshot1011:/mnt/dumpsdata/xmldatadumps/private/wikidatawiki/20241201$ pwd
/mnt/dumpsdata/xmldatadumps/private/wikidatawiki/20241201
xcollazo@snapshot1011:/mnt/dumpsdata/xmldatadumps/private/wikidatawiki/20241201$ tail -n 5 dumplog.txt 
2024-12-12 19:30:09: wikidatawiki (ID 433008) 5993 pages (1.0|314.8/sec all|curr), 280000 revs (48.9|52.5/sec all|curr), 99.8%|99.8% prefetched (all|curr), ETA 2026-06-07 11:26:03 [max 2286480581]
2024-12-12 19:30:09: wikidatawiki (ID 432995) 18901 pages (3.3|566.2/sec all|curr), 477000 revs (83.2|30.0/sec all|curr), 99.9%|99.9% prefetched (all|curr), ETA 2025-10-26 17:06:08 [max 2286480585]
2024-12-12 19:30:09: wikidatawiki (ID 433005) 10904 pages (1.9|1350.5/sec all|curr), 360000 revs (62.8|123.9/sec all|curr), 99.8%|99.8% prefetched (all|curr), ETA 2026-02-07 03:12:26 [max 2286480581]
2024-12-12 19:30:10: wikidatawiki (ID 433003) 11039 pages (1.9|1047.1/sec all|curr), 395000 revs (68.9|94.9/sec all|curr), 99.9%|99.9% prefetched (all|curr), ETA 2025-12-31 18:43:54 [max 2286480585]
2024-12-12 19:30:14: wikidatawiki (ID 433000) 3466 pages (0.6|152.7/sec all|curr), 243000 revs (42.4|44.0/sec all|curr), 99.9%|99.9% prefetched (all|curr), ETA 2026-08-29 12:18:40 [max 2286480584]

So the process killing did not re-start the dumps.

xcollazo added a comment.Dec 20 2024, 3:50 PM

Let's try and finish it. snapshot1011 will soon be busy with the 2024-12-20 partial run, so let's run it on snapshot1014:

sudo -u dumpsgen bash
cd /srv/deployment/dumps/dumps/xmldumps-backup

screen -S xcollazo-wikidatawiki-20241201-run

bash ./worker --date 20241201 --skipdone --exclusive --log --configfile /etc/dumps/confs/wikidump.conf.dumps:bigwikis --wiki wikidatawiki

If we finish successfully, we'd need to force rsync to the clouddumps. @Milimetric do we have how to force rsync documented somewhere?

xcollazo mentioned this in T382510: Wikimedia Dump Process seems to have stalled.Dec 20 2024, 3:52 PM
Ahoelzl added a comment.Jan 6 2025, 7:00 PM

@xcollazo can we consider this resolved?

xcollazo added a comment.Jan 6 2025, 7:14 PM
In T382084#10433765, @Ahoelzl wrote:

@xcollazo can we consider this resolved?

We still need to force rsync as per T382084#10418942.

@Milimetric could you share the steps?

• VirginiaPoundstone edited projects, added: Data-Engineering; removed: Data-Engineering (Q2 2024 October 1st - December 31th).Jan 6 2025, 8:40 PM
Ottomata closed this task as Resolved.Jan 23 2025, 3:30 PM
Ottomata claimed this task.

According to @Milimetric this is resolved.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits