Page MenuHomePhabricator

Include v2 dumps into mirrors rsync configuration
Closed, ResolvedPublic

Description

As we migrate certain v1 dumps to v2, we want to help mirrors make the transition automatically. We should update our rsync config so that mirrors include v2 dumps by default.

We'll communicate about this with mirrors, but still, let's also be careful not to suddenly push a large volume of data through this config, so our plan is to update the config to introduce a mix of v1 and v2 dumps at first.

Acceptance criteria

  • All mirrors that include in-scope v1 dumps begin to include a mix of v1 and v2, ideally without increasing overall file sizes. For example, dumpslastfive can become "the 3 latest v1 dumps, and 2 latest v2 dumps."

Event Timeline

Summary so far

Happy to report that to accomplish this, we do not need to touch the v1 dumps code or the v2 code. The mirror rsync modules and the lists that feed them are in operations/puppet, so we can make changes there and keep v1 and v2 decoupled.

Where the configuration is

  • modules/profile/templates/dumps/distribution/mirrors/rsyncd.conf.dumps_to_public.erb defines dumpslastfive, dumpslastfour, dumpslastthree, dumpslasttwo and dumpslastone. Each module reads include from = <xmldumpsdir>/rsync-inc-last-N.txt.
  • modules/dumps/files/web/list-last-good-dumps.sh runs every day at 03:55. It calls list-last-n-good-dumps.py, which writes rsync-inc-last-N.txt. The script scans public/<wiki>/<YYYYMMDD>/ and uses status.html to find good v1 runs.
  • v2 files are in public/other/mediawiki_content_{history,current}/<wiki>/<YYYY-MM-DD>/xml/{bzip2,7z}/. hdfs_rsync_job copies them from /wmf/data/exports with --delete. SHA256SUMS marks a finished directory.
  • The HDFS purge keeps v2 exports for 90 days. Thus the distribution hosts have 3 v2 runs at most today.

Sizes on clouddumps1001 (GiB)

Size that each module sends today:

ModuleGiB
dumpslastone10192
dumpslasttwo20402
dumpslastthree30542
dumpslastfour41389
dumpslastfive51158

One run, 2026-09-01:

PartGiB
v1, not deprecated (SQL tables, pages-articles*, multistream, abstracts, other)2023
v1, deprecated stubs (stub-meta-history, stub-meta-current, stub-articles)1607
v1, deprecated pages-meta-current725
v1, deprecated pages-meta-history bz24875
v1, deprecated pages-meta-history 7z962
v1 total10191
v2 mediawiki_content_history bz27200
v2 mediawiki_content_current bz2464
v2 7z (estimate, see below)1420

v2 7z has no output yet. The first 7z run is on 2026-10-01. The estimate uses the v1 ratio of 7z to bz2 for history (0.197).

v2 history bz2 is larger than v1 history bz2

v2 mediawiki_content_history bz2 is 7200 GiB. v1 pages-meta-history bz2 is 4875 GiB. v2 is 48% larger. We do not know the cause yet. We will examine the v2 bz2 output and consider whether to tune it to make smaller files. We prefer this to a mirror configuration without v2 7z.

Wiki coverage

The 2026-09-01 run has 1028 wikis in v1 and 859 wikis in v2. This is by design, as v2 only dumps currently active, public wikis. v1 continues to dump older closed wikis.

Proposed module content

Each module keeps the v1 files that are not deprecated for all N runs. In k of the runs, v2 replaces the deprecated v1 files. The deprecated v1 files stay in the other N - k runs.

size = N * (v1 not deprecated) + (N - k) * (v1 deprecated) + k * (v2)

We will remove all deprecated v1 files at a later date. Thus each module must have at least one v2 run (k >= 1). The 90 day retention gives k <= 3. (We could also bump the 90 day retention, however, we are running low on HDFS storage (see T437605) so this must be considered carefully.

Possible permutations, from the 2026-09-01 sizes:

Modulekbz2 only (GiB)Changebz2 + 7z (GiB)Change
dumpslastone19687-5.0%11108+9.0%
dumpslasttwo119878-2.6%21299+4.4%
dumpslasttwo219374-5.0%22215+8.9%
dumpslastthree130069-1.5%31490+3.1%
dumpslastthree229565-3.2%32406+6.1%
dumpslastthree329062-4.8%33323+9.1%
dumpslastfour140260-2.7%41681+0.7%
dumpslastfour239756-3.9%42597+2.9%
dumpslastfour339253-5.2%43514+5.1%
dumpslastfive150451-1.4%51872+1.4%
dumpslastfive249947-2.4%52788+3.2%
dumpslastfive349444-3.4%53705+5.0%

With bz2 only, all permutations are smaller than today. With 7z, all permutations are larger than today, although not by much.

Next steps

  • Find the cause of the larger v2 history bz2 files, and consider the cost of more aggressive bzip2 compression. If small, go for it.
  • Get the real v2 7z sizes after the 2026-10-01 run.
  • Choose k for each module.
  • Change operations/puppet: add a v2 list script, merge the v1 and v2 lists in list-last-good-dumps.sh, and update the rsyncd template. Test the rsync filter rules against a local rsyncd before deployment.

Update: v2 bz2 size

My previous comment said that v2 mediawiki_content_history bz2 is 48% larger than v1 pages-meta-history bz2. That is not correct. The compression is the same. The difference is one wiki.

Per-wiki comparison, 2026-09-01

  • 858 wikis are in both v1 and v2. For these wikis, v1 history bz2 is 4,986,422 MiB and v2 history bz2 is 4,989,720 MiB. The ratio v2/v1 is 1.001.
  • For the 669 wikis with at least 10 MiB of v1 history, the median ratio is 1.001. The 10th percentile is 0.999 and the 90th percentile is 1.002. The minimum is 0.971 and the maximum is 1.010.
  • One wiki is only in v2: wikidatawiki, with 2,327 GiB of history bz2. v1 stopped making the full history of wikidatawiki, because the job failed all the time. That work was done in T431267.

Thus v2 does not need a change to its bz2 output. Both v1 and v2 use bzip2 level 9, which is the maximum block size.

v2 directory check

I compared every v2 directory on clouddumps1001 with its SHA256SUMS. All files are in the manifest, no listed file is missing, and all file names have the dumper format. No failed run left files. Each directory also has a _SUCCESS file, which is not in the manifest. T438748 corrected this recently.

Corrected sizes, one run (2026-09-01, GiB)

PartGiB
v1, not deprecated2023
v1, deprecated (no wikidatawiki history)8168
v2 bz2: mediawiki_content_history 7200 (of which wikidatawiki is 2327) + mediawiki_content_current 4647664
v2 7z (estimate)770 to 960

wikidatawiki is on the 7z recompression_deny_list in the export DAG, so v2 7z has no wikidatawiki history. The low estimate uses v1 7z minus 20%, which is the gain that the DAG records for the v2 7z writer. The high estimate uses the v1 7z size. We get the real value after the 2026-10-01 run.

Baseline

Until T431267, the mirror modules included the wikidatawiki full history in each v1 run. The distribution hosts no longer have those runs, so the baseline below is an estimate:

  • v1 wikidatawiki history bz2: 2327 GiB. This is the v2 size. For the other 858 wikis, v2 bz2 is within 0.1% of v1.
  • v1 wikidatawiki history 7z: 459 GiB. This uses the v1 ratio of 7z to bz2 for the other wikis (0.197). wikidatawiki content is JSON, so the real ratio can be different.
  • The baseline adds these two values to each run in the module. If v1 did not make the wikidatawiki history for some runs, the real baseline was smaller.

Module size (GiB):

ModuleTodayWith wikidatawiki history (estimate)
dumpslastone1019212979
dumpslasttwo2040225975
dumpslastthree3054238901
dumpslastfour4138952535
dumpslastfive5115865091

Permutations (GiB)

k is the number of v2 runs in the module. The change is against the baseline with the wikidatawiki history.

Modulekbz2 onlyChangebz2 + 7z (low)Changebz2 + 7z (high)Change
dumpslastone19687-25.4%10457-19.4%10649-18.0%
dumpslasttwo119878-23.5%20648-20.5%20840-19.8%
dumpslasttwo219374-25.4%20913-19.5%21298-18.0%
dumpslastthree130069-22.7%30839-20.7%31031-20.2%
dumpslastthree229565-24.0%31104-20.0%31489-19.1%
dumpslastthree329062-25.3%31370-19.4%31947-17.9%
dumpslastfour140260-23.4%41030-21.9%41222-21.5%
dumpslastfour239756-24.3%41295-21.4%41680-20.7%
dumpslastfour339253-25.3%41561-20.9%42138-19.8%
dumpslastfive150451-22.5%51221-21.3%51413-21.0%
dumpslastfive249947-23.3%51486-20.9%51871-20.3%
dumpslastfive349444-24.0%51752-20.5%52329-19.6%

All permutations are 18% to 25% smaller than the baseline. The main reasons:

  • The stubs and pages-meta-current of each v1 run that v2 replaces are no longer sent. That is 2,331 GiB for each run.
  • v2 makes no 7z for wikidatawiki. The v1 7z copy was about 459 GiB for each run.

Against the size that the modules send today, without the wikidatawiki history, the change is between -5.2% and +4.6%.

Next steps

  • Get the real v2 7z sizes after the 2026-10-01 run.
  • Choose k for each module.
  • Change operations/puppet: add a v2 list script, merge the v1 and v2 lists in list-last-good-dumps.sh, and update the rsyncd template. Test the rsync filter rules against a local rsyncd before deployment.

Decision needed: how to choose between v1 and v2 for each run

This comment explains a behavior of the mirror change, the two rules we can use, and the cost of each. We recommend the second rule, but it needs longer retention for v2 files. We ask for a product decision before we continue.

Background

  • The mirror rsync modules (dumpslastone, dumpslasttwo, dumpslastthree, dumpslastfour, dumpslastfive) serve the last 1 to 5 monthly runs.
  • Each v1 run has files that are deprecated (stubs, pages-meta-current, pages-meta-history) and files that are not deprecated (SQL tables, pages-articles, abstracts and others).
  • In a run that v2 replaces, the module serves the files that are not deprecated from v1, and the content from v2. It does not serve the deprecated v1 files.
  • v1 and v2 both run on the 1st of the month. The v1 run of a month can be complete some days before the v2 export of the same month is on the distribution hosts.
  • Mirrors copy with rsync. When a file leaves a module, most mirrors delete their copy.

Rule 1: a fixed mix (for example, the last 2 of 5 runs are v2)

With this rule, the number of v2 runs in each module stays the same. The problem is that a run changes from v1 to v2, or from v2 to v1, while it is in the module:

  • When the v1 run of a new month is complete before its v2 export, the run is one of the "last 2", but it has no v2 yet. The module must either serve the full v1 run, or serve nothing for it. If it serves the full v1 run, mirrors copy about 8,200 GiB of deprecated files, and they delete them some days later when v2 arrives.
  • Each month, all runs move one position toward the oldest. With "the last 2 of 5 are v2", a run is v2 in positions 1 and 2, and v1 in positions 3 to 5. So every run changes from v2 to v1 when it moves to position 3. Mirrors then delete about 7,700 to 8,600 GiB of v2 files and copy about 8,200 GiB of deprecated v1 files for that run.

Thus a fixed mix makes mirrors copy and delete a large quantity of data each month.

Rule 2: a cutoff date (recommended)

We set one cutoff date for all modules:

  • A run with a date on or after the cutoff is always v2. The module never serves its deprecated v1 files. Until the v2 export of that run is on the distribution hosts, the module serves only the v1 files that are not deprecated.
  • A run with a date before the cutoff is always v1. The module serves all of its files.

With this rule, no run changes from one version to the other. Each month, one more run in each module is v2, until all runs are v2. This is the same direction as the sunset of the deprecated v1 files.

Example with cutoff 2026-10-01. The table shows the number of v2 runs in each module:

Monthdumpslastonedumpslasttwodumpslastthreedumpslastfourdumpslastfive
2026-1011111
2026-1112222
2026-1212333
2027-0112344
2027-0212345

Size of dumpslastfive at each step (GiB, from the 2026-09-01 sizes, with the 7z estimate of 770 to 960 GiB for each v2 run):

v2 runsSizeChange vs today (51158)Change vs baseline with wikidatawiki history (65091)
151221 to 51413+0.1% to +0.5%-21.3% to -21.0%
251486 to 51871+0.6% to +1.4%-20.9% to -20.3%
351752 to 52329+1.2% to +2.3%-20.5% to -19.6%
452017 to 52787+1.7% to +3.2%-20.1% to -18.9%
552283 to 53244+2.2% to +4.1%-19.7% to -18.2%

Caveat: rule 2 needs longer retention for v2 files

The HDFS purge keeps v2 exports for 90 days (data_purge.pp). hdfs_rsync_job copies them to the distribution hosts with --delete, so the hosts also keep only 3 v2 runs. With rule 2, dumpslastfour needs 4 v2 runs after 4 months, and dumpslastfive needs 5 v2 runs after 5 months. With 90 days of retention, those runs are not available.

There are two options:

  • Option A: increase the v2 retention to 5 monthly runs (about 150 days). This adds 2 v2 runs, which is 16,900 to 17,300 GiB on HDFS (before HDFS replication), and the same on each distribution host (clouddumps1001 and clouddumps1002).
  • Option B: keep 90 days. When dumpslastfour and dumpslastfive have 3 v2 runs, they stop at 3 runs. Mirrors that use these modules then get fewer runs than the module name says.

We recommend option A.

Questions for product

  1. Do you agree with rule 2 (cutoff date)?
  2. Which cutoff date do you want? 2026-09-01 is available now. 2026-10-01 is the first run with v2 7z files.
  3. Do you agree with option A (5 months of v2 retention)?

Excellent analysis, thank you!

I'd like to see @HCoplin-WMF 's thoughts but my answers are:

Do you agree with rule 2 (cutoff date)?

Yes, this seems simpler and more efficient.

Which cutoff date do you want? 2026-09-01 is available now. 2026-10-01 is the first run with v2 7z files.

I'd put this at 2026-11-01. In general, I like to announce things at least a week before they happen. As I understand it, if we set it to October, mirrors would immediately get the v2 dumps at the same time as the comms go out.

Do you agree with option A (5 months of v2 retention)?

Yes, it seems most mirrors today are on either dumpslastfive and dumpslasttwo so I'd rather not change dumpslastfive along with the other changes.

Change #1346119 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] analytics: refinery: Keep File Exports for 170 days

https://gerrit.wikimedia.org/r/1346119

Update: v2 retention

Product approved the cutoff rule, the cutoff date 2026-11-01, and a longer v2 retention in T437463#12372843.

The retention must be 170 days, not the 150 days in my previous comment:

  • list-last-n-good-dumps.py lists the v1 run that is in progress, in addition to the 5 complete runs. So dumpslastfive needs the oldest v2 run for about 3 more weeks, until the next v1 run is complete.
  • The purge runs only on the 1st of the month. The oldest v2 run is 150 to 153 days old at one purge, and 181 to 184 days old at the next purge. 170 days keeps it at the first purge and deletes it at the second.

Cost

HDFS and each distribution host keep 6 v2 runs for most of the month. Today they keep 3. One v2 run is 8,434 to 8,626 GiB: 7,664 GiB of bz2, and 770 to 960 GiB of 7z (estimate).

LocationMore storage
HDFS, logical27.2 to 27.8 TB
HDFS, on-disk (replication 3)81.5 to 83.4 TB
clouddumps100125,300 to 25,900 GiB
clouddumps100225,300 to 25,900 GiB

This new HDFS cost is quite significant, and it goes against the work we are trying to do in T437605: Figure out the biggest HDFS consumers over time. However, we do want to be able to sunset the v1 XML dumps in the near future so we have to pay the cost.

For mid term, I am opening T439733: File Exports: Consider storing the exports in Ceph instead of HDFS to consider moving the exports from HDFS and into Ceph.

Distribution hosts

The distribution hosts do not have this free space today:

Host/srv/dumps usedFree
clouddumps1001161T (87%)26T
clouddumps1002159T (86%)27T

They need about 25.3 TiB for the 3 more v2 runs, and about 2.8 TiB for the 7z of the 3 v2 runs that they keep today.

v1 used to alternate a full run on the 1st and a partial run without pages-meta-history on the 20th. Now every v1 run is a full run, and the hosts keep 7 to 10 of them. So we copy the old pattern: on the distribution hosts, only the 3 newest v1 runs keep pages-meta-history. The older runs keep all other files, and each wiki keeps the same number of runs as before.

  • A dry run on clouddumps1001 removes 131,360 pages-meta-history files (27.16 TiB), and no other file.
  • /srv/dumps goes to about 72% after the cleanup, and to about 87% when all the v2 storage is there (March 2027).
  • The mirror modules serve whole v1 runs before the cutoff. When one of these runs is older than the 3 newest runs, its pages-meta-history files leave the hosts, and the mirrors delete them too.

After the cutoff, the mirror modules do not send the other deprecated v1 files (stubs, pages-meta-current), but v1 still writes them, and the web downloads still serve them. The distribution hosts get more storage back when v1 stops writing the deprecated files.

Changes

The deploy order is important:

  1. dumps: Keep pages-meta-history only in the 3 newest runs on replicas changes $keep_replicas in modules/dumps/manifests/web/cleanups/xmldumps.pp to hugewikis.dblist:3:4, bigwikis.dblist:3:5 and default:3:7, with a patterns file that keeps everything except pages-meta-history.
  2. analytics: refinery: Keep File Exports for 170 days sets --older-than='170' for the two File Export purge jobs in data_purge.pp. A dry run of refinery-drop-older-than gives the same --execute checksums as the change.
  3. The mirror list changes, after 1 and 2.

Change #1346629 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] dumps: Keep pages-meta-history only in the 3 newest runs on replicas

https://gerrit.wikimedia.org/r/1346629

Change #1346119 abandoned by Xcollazo:

[operations/puppet@production] analytics: Keep File Exports for 170 days

Reason:

After reviews, it is now clear to me this is not a good approach.

https://gerrit.wikimedia.org/r/1346119

Change #1346752 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] dumps: Keep File Exports on the distribution hosts, not in HDFS

https://gerrit.wikimedia.org/r/1346752

Update: v2 retention, second approach

After the review of the 170-day change (now abandoned), the extra v2 runs stay on the distribution hosts only, not in HDFS:

  • The two File Export fetches run without --delete. cleanup_old_miscdumps.sh keeps the 6 newest runs of each wiki on clouddumps1001 and clouddumps1002.
  • HDFS keeps File Exports for 45 days (2 runs) in place of 90 days (3 runs). This frees about 27 TB on-disk in HDFS. The 170-day plan would have added about 83 TB.
  • The cost on the distribution hosts does not change. With dumps: Keep pages-meta-history only in the 3 newest runs on replicas, they end at about 87% of /srv/dumps.
  • Known risk: without --delete, a failed export attempt can leave files on the hosts that are not in SHA256SUMS. T439759 removes this risk.

Change #1346764 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] dumps: Write mirror rsync lists that mix v1 dumps and File Exports

https://gerrit.wikimedia.org/r/1346764

Change #1346765 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] dumps: Use the mixed rsync lists in the mirror modules

https://gerrit.wikimedia.org/r/1346765

Which cutoff date do you want? 2026-09-01 is available now. 2026-10-01 is the first run with v2 7z files.

I'd put this at 2026-11-01. In general, I like to announce things at least a week before they happen. As I understand it, if we set it to October, mirrors would immediately get the v2 dumps at the same time as the comms go out.

After chatting with Halley, I'm thinking November might be too early a cutoff. There's a good chance we'll need to spend October and November dealing with lingering data issues or unexpected concerns, so in a way it's good for us to migrate mirrors later to avoid them experiencing the same issues with their users.

Would we be able to easily move the cutoff to December 2026 instead?

Would we be able to easily move the cutoff to December 2026 instead?

Yes, easy fix. Will do.

Change #1346752 merged by Btullis:

[operations/puppet@production] dumps: Keep File Exports on the distribution hosts, not in HDFS

https://gerrit.wikimedia.org/r/1346752

Change #1352580 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] dumps: Remove the generator settings from the web cleanups

https://gerrit.wikimedia.org/r/1352580

Change #1352580 merged by Brouberol:

[operations/puppet@production] dumps: Remove the generator settings from the web cleanups

https://gerrit.wikimedia.org/r/1352580

Would we be able to easily move the cutoff to December 2026 instead?

Yes, easy fix. Will do.

For completeness, 2026-12-01 is now set at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1346764/5/modules/dumps/files/web/list-last-good-dumps.sh.

Change #1346629 merged by Btullis:

[operations/puppet@production] dumps: Keep pages-meta-history only in the 3 newest runs on replicas

https://gerrit.wikimedia.org/r/1346629

Update: changes merged

  • https://gerrit.wikimedia.org/r/1346752 (2026-10-07): the distribution hosts keep the 6 newest File Export runs. HDFS keeps them for 45 days. The first HDFS purge with 45 days is on 2026-11-01.
  • https://gerrit.wikimedia.org/r/1352580 (2026-10-08): removes unused cleanup settings. No change on the hosts.
  • https://gerrit.wikimedia.org/r/1346629 (2026-10-09): only the 3 newest v1 runs keep pages-meta-history. The first cleanup ran on 2026-10-10. Both hosts are now at 130 TiB used (70%) of /srv/dumps, from 161 TiB and 159 TiB on 2026-09-30. Each wiki keeps the same number of runs.

Open

Change #1346764 merged by Brouberol:

[operations/puppet@production] dumps: Write mirror rsync lists that mix v1 dumps and File Exports

https://gerrit.wikimedia.org/r/1346764

Change #1354906 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] dumps: Use exclude rules in the mixed mirror rsync lists

https://gerrit.wikimedia.org/r/1354906

Update: rsync list size checked early

https://gerrit.wikimedia.org/r/1346764 merged on 2026-10-13. Before the cutoff, its lists are the same as today's lists, so mirrors see no change yet.

My last update planned a list size check in November. I did it now on clouddumps1001, with a test list that uses --cutoff 2026-10-01 (one replaced run) and a test rsync daemon. I timed one rsync -r --list-only of dumpslastfive:

ListLinesDaemon CPU (s)
Today (rsync-inc-last-5.txt)11,82769
1346764 as merged121,812about 700
https://gerrit.wikimedia.org/r/135490612,11478

The merged lists name each file of a replaced run, and that is too slow. 1354906 uses 5 exclude rules for each run date in place of that. It serves the same files, at about 1.13 times the CPU of today.

Open

Change #1354906 merged by Brouberol:

[operations/puppet@production] dumps: Use exclude rules in the mixed mirror rsync lists

https://gerrit.wikimedia.org/r/1354906

Change #1346765 merged by Brouberol:

[operations/puppet@production] dumps: Use the mixed rsync lists in the mirror modules

https://gerrit.wikimedia.org/r/1346765

Change #1346765 merged by Brouberol:

[operations/puppet@production] dumps: Use the mixed rsync lists in the mirror modules

https://gerrit.wikimedia.org/r/1346765

With this patch merged, the changes are now effective in prod. This will start transitioning mirrors to v2 from December 2026 and on.