| brouberol | |
| Mar 24 2025, 11:26 AM |
| F59725725: image.png | |
| May 6 2025, 9:54 PM |
The dumps v1 jobs will write their files on a CephFS volume. However, that volume is not publicly accessible on the internet, meaning that the files will need to be synchronized onto the clouddumps100[1-2].wikimedia.org hosts, to /srv/dumps/xmldatadumps/public/.
We probably need to setup a way to run some kind of continuous synchronization.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| dumps v1: Update sync tmpdir on clouddumps hosts | repos/data-engineering/airflow-dags!1313 | btullis | update_dumps_tmpdir | main | |
| [dumps/v1] Include the whole public/${wiki} directory during sync tasks | repos/data-engineering/airflow-dags!1293 | btullis | sync_complete_wiki_dir | main | |
| Add GNU parallel and pssh to the list of available tools | repos/data-engineering/sync-utils!4 | btullis | add_ssh | main | |
| test_k8s/dumpsv1: interweave dump and sync tasks | repos/data-engineering/airflow-dags!1283 | brouberol | T389784 | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | BTullis | T389784 Synchronize the dumps files to the clouddumps hosts | |||
| Resolved | BTullis | T390736 Create a container image containing the necessary sync utilities for publishing dump files | |||
| Resolved | BTullis | T390738 [dumps] Configure rsyncd authentication - or a suitable alternative |
Let's look at how things are done at the moment, then we can work out how best to replicate this functionality and which bits we might be able to improve:
Firstly, I don't think that any synchronization happens on the snapshot servers, themselves. It happens on the dumpsdata servers, using these puppet classes I believe.
They set up a systemd timer called dumps-rsyncer.service on each of the dumpsdata hosts, which use the templates here:
From this command:
btullis@cumin1002:~$ sudo cumin 'A:dumps and not D{htmldumper1001.eqiad.wmnet}' 'systemctl show dumps-rsyncer.service|grep ExecStart='...we can see that this service is only currently deployed to:
dumpsdata1006 is running this command:
/usr/local/bin/rsync-via-primary.sh --do_tarball --do_rsync_xml --xmldumpsdir /data/xmldatadumps/public --xmlremotedirs dumpsdata1007.eqiad.wmnet::data/xmldatadumps/public/,clouddumps1001.wikimedia.org::data/xmldatadumps/public/,clouddumps1002.wikimedia.org::data/xmldatadumps/public/
dumpsdata1003 is running this command:
/usr/local/bin/rsync-via-primary.sh --do_rsync_misc --do_rsync_miscsubs --miscdumpsdir /data/otherdumps --miscremotedirs clouddumps1001.wikimedia.org::data/xmldatadumps/public/other/,clouddumps1002.wikimedia.org::data/xmldatadumps/public/other/ --miscsubdirs incr,categoriesrdf --miscremotesubs dumpsdata1007.eqiad.wmnet::data/otherdumps/
In addition these, dumpsdata1007 has the profile::dumps::generation::server::dumpstatusfiles_sync profile applied.
This profile adds the following class: dumps::web::dumpstatusfiles
That causes unpack-dumpstatusfiles.sh to be executed every 5 minutes on the XML fallback host.
I think that this setup is so that the HTML status files can be kept up to date independently of the larger dumps files, but also so that the index.html files don't show links to fines that haven't been copied from dumpsdata servers to clouddumps servers yet. This commit and T179857: Make sure rsynced dump status/html files don't contain links to files not yet copied over explain the intention of this a little more.
It's pretty convoluted. I wonder how much we can remove.
Ah, now I see that it is not only dumpsdata1007 that unpacks these HTML status files. The two clouddumps servers also run the same script.
btullis@cumin1002:~$ sudo cumin C:dumps::web::dumpstatusfiles 3 hosts will be targeted: clouddumps[1001-1002].wikimedia.org,dumpsdata1007.eqiad.wmnet DRY-RUN mode enabled, aborting
We can see from here that the /usr/local/bin/rsync-via-primary.sh script specifically excludes *.html and *,json files.
This means that these HTML and JSON files are only updated on the clouddumps hosts by means of the following processes:
Change #1135416 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Add an rsync fragment to permit dse-k8s pods to sync mediawiki dumps
Change #1135416 merged by Btullis:
[operations/puppet@production] Add an rsync fragment to permit dse-k8s pods to sync mediawiki dumps
Change #1140664 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Update dumpsgen SSH settings on clouddumps servers.
Change #1140664 merged by Btullis:
[operations/puppet@production] Update dumpsgen SSH settings on clouddumps servers.
It's working and the first rsync over ssh is running now.
I am running this command in the pod.
runuser@sync-pod-with-cephfs-volume:~$ rsync -av /mnt/dumpsdata/xmldatadumps/public dumpsgen@clouddumps1001.wikimedia.org:/srv/mediawiki-dumps-legacy/xmldatadumps
On the server, this is the process tree for the dumpsgen user.
btullis@clouddumps1001:/srv/mediawiki-dumps-legacy$ pstree -a dumpsgen
sshd
└─rsync --server -vlogDtpre.iLsfxCIvu . /srv/mediawiki-dumps-legacy/xmldatadumps
└─rsync --server -vlogDtpre.iLsfxCIvu . /srv/mediawiki-dumps-legacy/xmldatadumps
systemd --user
└─(sd-pam)I had to get rid of the ChrootDirectory in order to use rsync because it doesn't exist in the chroot.
I am still considering whether to use just the SFTP protocol and use rclone copy with an sftp back-end.
But for now, rsync over ssh is working.
Change #1141926 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] mediawiki-dumps-legacy: rename Secret key associated to private key
Change #1141926 merged by Brouberol:
[operations/deployment-charts@master] mediawiki-dumps-legacy: rename Secret key associated to private key
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/sync-utils/-/merge_requests/4
Add GNU parallel and pssh to the list of available tools
I am now having some success with the command:
runuser@sync-pod-with-cephfs-volume:/home/sync-utils$ parallel-rsync -v -ra -H dumpsgen@clouddumps1001.wikimedia.org -H dumpsgen@clouddumps1002.wikimedia.org /mnt/dumpsdata/xmldatadumps/public/ /srv/mediawiki-dumps-legacy/xmldatadumps/public
Transfer speeds are around 190 MB/s when the pod has 2 CPUs and 4GB of RAM. I think that this is workable.
We can populate a file to use instead of the two -H options, which would make the command line a little nicer.
We can also use a /home/runuser/.ssh/config file to set the default cipher to be a slightly faster one, or pass other ssh options with a -S argument, or other rsync options with a -X option.
Change #1143060 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/deployment-charts@master] Add ssh_config and an rsync_targets files to mediawiki-dumps-legacy
Change #1143060 merged by jenkins-bot:
[operations/deployment-charts@master] Add ssh_config and an rsync_targets files to mediawiki-dumps-legacy
Change #1143101 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] mediawiki-dumps-legacy: fix typo
Change #1143103 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/deployment-charts@master] Fix typo in mediawiki-dumps-legacy
Change #1143103 abandoned by Btullis:
[operations/deployment-charts@master] Fix typo in mediawiki-dumps-legacy
Reason:
duplicate
Change #1143101 merged by Brouberol:
[operations/deployment-charts@master] mediawiki-dumps-legacy: fix typo
Change #1143106 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] Fix typo
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1283
test_k8s/dumpsv1: interweave dump and sync tasks
There is a follow-up ticket here: T393687: Skip sync tasks for private wikis, or allow them to fail gracefully
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1293
[dumps/v1] Include the whole public/${wiki} directory during sync tasks
Change #1145201 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Update the managed airflow temp directory for dumps on k8s
Change #1145201 merged by Btullis:
[operations/puppet@production] Update the managed airflow temp directory for dumps on k8s
btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1313
dumps v1: Update sync tmpdir on clouddumps hosts
Change #1145874 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] clouddumps: Manage directories beneath /srv/dumps/xmldatadumps_airflow_temp
Change #1145874 merged by Btullis:
[operations/puppet@production] clouddumps: Manage directories beneath /srv/dumps/xmldatadumps_airflow_temp