WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	BTullis
	Dec 19 2024, 11:52 AM

Description

As per the APP FY24-25: OKRs and Hypotheses spreadsheet:

If the beta cluster is configured to run MediaWiki with PHP 8.1 then the Data Platform Engineering group and their SRE team will be able to validate whether the existing dumps code functions correctly, or whether any significant functional changes would be required.

We have now had confirmation that the beta cluster has been upgraded to PHP version 8.1, so the Data-Platform-SRE team can begin this validation process.

btullis@deployment-snapshot05:~$ php -v
PHP 8.1.31 (cli) (built: Nov 21 2024 21:07:42) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.31, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.31, Copyright (c), by Zend Technologies

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Upgrade PHP on the misc dumps worker - snapshot1016	operations/puppet	production	+2 -1
Upgrade snapshot hosts to PHP version 8.1 - except snapshot1016	operations/puppet	production	+15 -1
dumps: Use proc_close() to close proc_open() subprocess	mediawiki/core	REL1_43	+1 -1
dumps: Use proc_close() to close proc_open() subprocess	mediawiki/core	REL1_42	+1 -1
dumps: Use proc_close() to close proc_open() subprocess	mediawiki/core	REL1_39	+1 -1
dumps: Use proc_close() to close proc_open() subprocess	mediawiki/core	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T389515: PHP Deprecated: Return type of Flow\Search\Iterators\AbstractIterator::current() should either be compatible with Iterator::current(): mixed, or the #[\ReturnTypeWillChange] attribute should be used
Mentioned Here: P72364 T382484 - dump output file listing

Event Timeline

BTullis created this task.Dec 19 2024, 11:52 AM

Restricted Application added a project: Data-Platform. · View Herald TranscriptDec 19 2024, 11:52 AM

bking subscribed.Dec 19 2024, 2:33 PM

BTullis renamed this task from WE 5.4 KR - (Hypothesis TBD) - Q3 FY24/55 - Validate Dumps 1.0 compatibility with PHP 8.1 to WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/55 - Validate Dumps 1.0 compatibility with PHP 8.1.Dec 19 2024, 5:22 PM

I'm running the first manual dump with php8.1 on the beta cluster now.

dumpsgen@deployment-snapshot05:/srv/deployment/dumps/dumps/xmldumps-backup$ bash ./worker --date last --skipdone --exclusive --log --configfile /etc/dumps/confs/wikidump.conf.labs --wiki meta
python3 ./worker.py --configfile /etc/dumps/confs/wikidump.conf.labs --log --skipdone --exclusive --date last meta
Running meta...
2024-12-20 11:35:56: meta Creating /mnt/dumpsdata/xmldatadumps/private/meta/20241220 ...
2024-12-20 11:35:56: meta Creating /mnt/dumpsdata/xmldatadumps/public/meta/20241220 ...

Failed with:

dumps.exceptions.BackupError: command ''/usr/bin/php8.1' /srv/mediawiki/multiversion/MWScript.php getReplicaServer.php --wiki='meta' --group=dump' failed with return code 255  and error 'Fatal error: no version entry for `meta`.
 in /srv/mediawiki/multiversion/MWMultiVersion.php on line 696
'
Dump of wiki meta failed.

It's quite possible that I'm doing it incorrectly, so I'll have a closer look.

Yes, it was a pebcak.
I ran it again with this command:

bash ./worker --date last --skipdone --exclusive --log --configfile /etc/dumps/confs/wikidump.conf.labs --wiki metawiki

It's proceeding much more now.

I have seen a failure from one task, which is related to flow.

MWUnknownContentModelException from line 191 of /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php: The content model 'flow-board' is not registered on this wiki.

...but I think it's still proceeding, according to /mnt/dumpsdata/xmldatadumps/private/metawiki/20241220/dumplog.txt

BTullis triaged this task as High priority.Dec 20 2024, 11:56 AM

So the backup was a partial success. This error prevented the successful export of all stubs, which then caused dome jobs to be skipped later.

command /usr/bin/python3 xmlstubs.py --config /etc/dumps/confs/wikidump.conf.labs --wiki metawiki --articles /mnt/dumpsdata/xmldatadumps/public/metawiki/20241220/metawiki-20241220-stub-articles.xml.gz.inprog --h
istory /mnt/dumpsdata/xmldatadumps/public/metawiki/20241220/metawiki-20241220-stub-meta-history.xml.gz.inprog --current /mnt/dumpsdata/xmldatadumps/public/metawiki/20241220/metawiki-20241220-stub-meta-current.xm
l.gz.inprog (3386692) started...
2024-12-20 11:49:20: metawiki (ID 3386741) 579 pages (2001.1|2001.1/sec all|curr), 1000 revs (3456.1|3456.1/sec all|curr), ETA 2024-12-20 11:49:35 [max 51679]
2024-12-20 11:49:20: metawiki (ID 3386741) 1366 pages (2632.0|5948.2/sec all|curr), 2000 revs (3853.6|4354.5/sec all|curr), ETA 2024-12-20 11:49:33 [max 51679]
MWUnknownContentModelException from line 191 of /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php: The content model 'flow-board' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(246): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('flow-board', NULL)
#1 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(180): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('flow-board')
#2 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(92): MediaWiki\Content\ContentHandlerFactory->createForModelID('flow-board')
#3 /srv/mediawiki/php-master/includes/export/XmlDumpWriter.php(472): MediaWiki\Content\ContentHandlerFactory->getContentHandler('flow-board')
#4 /srv/mediawiki/php-master/includes/export/XmlDumpWriter.php(400): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 1)
#5 /srv/mediawiki/php-master/includes/export/WikiExporter.php(541): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-master/includes/export/WikiExporter.php(479): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-master/includes/export/WikiExporter.php(316): WikiExporter->dumpPages('page_id >= 1 AN...', false)
#8 /srv/mediawiki/php-master/includes/export/WikiExporter.php(211): WikiExporter->dumpFrom('page_id >= 1 AN...', false)
#9 /srv/mediawiki/php-master/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(1, 5001, false)
#10 /srv/mediawiki/php-master/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(703): DumpBackup->execute()
#12 /srv/mediawiki/php-master/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(156): require_once('/srv/mediawiki/...')
#14 {main}
nonzero return 1 from command '/usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=metawiki --dbgroupdefault=dump --full --stub --report=1000 --output=file:/mnt/dumpsdata/xmldatadumps/
temp/m/metawiki/metawiki-20241220-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/metawiki/metawiki-20241220-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file
:/mnt/dumpsdata/xmldatadumps/temp/m/metawiki/metawiki-20241220-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=1 --skip-footer --end 5001'

I will look into this.

I tried again with a different wiki, this time one which is listed in /srv/mediawiki/dblists/flow-labs.dblist

bash ./worker --date last --skipdone --exclusive --log --configfile /etc/dumps/confs/wikidump.conf.labs --wiki testwiki

This time we got a different error.

Spawning database subprocess: '/usr/bin/php8.1' '/srv/mediawiki/php-master/../multiversion/MWScript.php' 'fetchText.php' '--wiki' 'testwiki'
2024-12-20 12:33:09: testwiki (ID 3389767) 999 pages (222.0|222.0/sec all|curr), 1000 revs (222.3|222.3/sec all|curr), ETA 2024-12-20 12:33:23 [max 4044]
[48d6087fb84c3764571bd312] [no req]   TypeError: pclose(): supplied resource is not a valid stream resource
Backtrace:
from /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php(811)
#0 /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php(811): pclose(resource (process))
#1 /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php(271): MediaWiki\Maintenance\TextPassDumper->closeSpawn()
#2 /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php(201): MediaWiki\Maintenance\TextPassDumper->dump(bool)
#3 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(703): MediaWiki\Maintenance\TextPassDumper->execute()
#4 /srv/mediawiki/php-master/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#5 /srv/mediawiki/multiversion/MWScript.php(156): require_once(string)
#6 {main}
returned from 3389767 with 255

Its parent process was this (although it appears truncated):

command /usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpTextPass.php --wiki=testwiki --stub=gzip:/mnt/dumpsdata/xmldatadumps/public/testwiki/20241220/testwiki-20241220-stub-meta-current.xml.gz  --db
groupdefault=dump --report=1000 --spawn=/usr/bin/php8.1 --output=bzip2:/mnt/dumpsdata/xmldatadumps/public/testwiki/20241220/testwiki-20241220-pages-meta-current.xml.bz2.inprog --current

We saw several of the:

TypeError: pclose(): supplied resource is not a valid stream resource

However, this time it looks like the flow related tables have backed up successfully.

command /usr/bin/python3 xmlflow.py --config /etc/dumps/confs/wikidump.conf.labs --wiki testwiki --outfile /mnt/dumpsdata/xmldatadumps/public/testwiki/20241220/testwiki-20241220-flow.xml.bz2.inprog (3389822) sta
rted...
returned from 3389822 with 0
2024-12-20 12:34:06: testwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/testwiki/latest ...
2024-12-20 12:34:06: testwiki Adding symlink /mnt/dumpsdata/xmldatadumps/public/testwiki/latest/testwiki-latest-flow.xml.bz2 -> ../20241220/testwiki-20241220-flow.xml.bz2
2024-12-20 12:34:06: testwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/testwiki/latest ...
2024-12-20 12:34:06: testwiki adding rss feed file /mnt/dumpsdata/xmldatadumps/public/testwiki/latest/testwiki-latest-flow.xml.bz2-rss.xml
2024-12-20 12:34:06: testwiki Checksumming testwiki-20241220-flow.xml.bz2 via md5
2024-12-20 12:34:06: testwiki Checksumming testwiki-20241220-flow.xml.bz2 via sha1
command /usr/bin/python3 xmlflow.py --config /etc/dumps/confs/wikidump.conf.labs --wiki testwiki --outfile /mnt/dumpsdata/xmldatadumps/public/testwiki/20241220/testwiki-20241220-flowhistory.xml.bz2.inprog --hist
ory (3389871) started...
returned from 3389871 with 0
returned from 3389871 with 0
2024-12-20 12:34:46: testwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/testwiki/latest ...
2024-12-20 12:34:46: testwiki Adding symlink /mnt/dumpsdata/xmldatadumps/public/testwiki/latest/testwiki-latest-flowhistory.xml.bz2 -> ../20241220/testwiki-20241220-flowhistory.xml.bz2
2024-12-20 12:34:46: testwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/testwiki/latest ...
2024-12-20 12:34:46: testwiki adding rss feed file /mnt/dumpsdata/xmldatadumps/public/testwiki/latest/testwiki-latest-flowhistory.xml.bz2-rss.xml
2024-12-20 12:34:46: testwiki Checksumming testwiki-20241220-flowhistory.xml.bz2 via md5
2024-12-20 12:34:46: testwiki Checksumming testwiki-20241220-flowhistory.xml.bz2 via sha1

• VirginiaPoundstone added a project: Data-Engineering.Jan 6 2025, 8:40 PM

BTullis renamed this task from WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/55 - Validate Dumps 1.0 compatibility with PHP 8.1 to WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1.Jan 7 2025, 4:11 PM

BTullis edited projects, added Data-Platform-SRE; removed Data-Platform, Data-Platform-SRE (2024.11.30 - 2024.12.20).Jan 9 2025, 12:31 PM

BTullis moved this task from Incoming to Quarterly Goals on the Data-Platform-SRE board.

I have made progress on this by circumventing the failure path, but I don't know whether this is a sustainable fix.
Therefore, I think that need to reach out to engineers with PHP knowledge to understand whether this is the right fix, or whether we need to look into it more closely.

The error that was preventing the dumps was this.

TypeError: pclose(): supplied resource is not a valid stream resource

Backtrace:
from /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php(811)
#0 /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php(811): pclose(resource (process))
#1 /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php(271): MediaWiki\Maintenance\TextPassDumper->closeSpawn()
#2 /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php(201): MediaWiki\Maintenance\TextPassDumper->dump(bool)
#3 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(703): MediaWiki\Maintenance\TextPassDumper->execute()

I tried various approaches to increase the debugging, but I found that if I simply comment out these three lines from /srv/mediawiki/php-master/maintenance/includes/TextPassDumper.php, the dump completes successfully.

dumpsgen@deployment-snapshot05:/srv/deployment/dumps/dumps/xmldumps-backup$ ./worker --date 20250119 --skipdone --exclusive --log --configfile /etc/dumps/confs/wikidump.conf.labs --wiki testwiki
python3 ./worker.py --configfile /etc/dumps/confs/wikidump.conf.labs --log --skipdone --exclusive --date 20250119 testwiki
Running testwiki...
2025-01-16 11:15:34: testwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/private/testwiki/20250119 ...
2025-01-16 11:15:34: testwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/testwiki/20250119 ...
2025-01-16 11:15:34: testwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/testwiki/20250119 ...
2025-01-16 11:15:34: testwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/private/testwiki/20250119 ...
2025-01-16 11:15:34: testwiki Cleaning up old dumps for testwiki
2025-01-16 11:15:34: testwiki No old public dumps to purge.
2025-01-16 11:15:34: testwiki No old private dumps to purge.
2025-01-16 11:15:34: testwiki Starting backup of testwiki
<snip snip>
2025-01-16 11:23:24: testwiki SUCCESS: done.
Dump of wiki testwiki succeeded.

n.b. I haven't checked the data correctness of the dumps yet, but all stages seemed to run correctly and the timings and stdout seemed broadly correct.

I can tell that this is something to do with closing down spawned processes that are specifically for the dumps, but I don't know how to delve into it more deeply than that.

Plus, the mediawiki installation at /srv/mediawiki keeps on reverting my change every few minutes. I haven't found out how to keep this override in place.

I have reached out on the #mediawiki-engineering-teams Slack channel, to ask for help in understanding the implications of this workaround:
https://wikimedia.slack.com/archives/C05D7UDJZ5E/p1737028488390569

Change #1111958 had a related patch set uploaded (by Máté Szabó; author: Máté Szabó):

[mediawiki/core@master] dumps: Use proc_close() to close proc_open() subprocess

https://gerrit.wikimedia.org/r/1111958

gerritbot added a project: Patch-For-Review.Jan 16 2025, 12:08 PM

• mszabo subscribed.Jan 16 2025, 1:09 PM

Change #1111958 merged by jenkins-bot:

[mediawiki/core@master] dumps: Use proc_close() to close proc_open() subprocess

https://gerrit.wikimedia.org/r/1111958

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2025, 3:32 PM

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.13; 2025-01-21).Jan 16 2025, 4:01 PM

Change #1112317 had a related patch set uploaded (by Krinkle; author: Máté Szabó):

[mediawiki/core@REL1_43] dumps: Use proc_close() to close proc_open() subprocess

https://gerrit.wikimedia.org/r/1112317

Change #1112318 had a related patch set uploaded (by Krinkle; author: Máté Szabó):

[mediawiki/core@REL1_39] dumps: Use proc_close() to close proc_open() subprocess

https://gerrit.wikimedia.org/r/1112318

Change #1112319 had a related patch set uploaded (by Krinkle; author: Máté Szabó):

[mediawiki/core@REL1_42] dumps: Use proc_close() to close proc_open() subprocess

https://gerrit.wikimedia.org/r/1112319

Change #1112318 merged by jenkins-bot:

[mediawiki/core@REL1_39] dumps: Use proc_close() to close proc_open() subprocess

https://gerrit.wikimedia.org/r/1112318

Change #1112319 merged by jenkins-bot:

[mediawiki/core@REL1_42] dumps: Use proc_close() to close proc_open() subprocess

https://gerrit.wikimedia.org/r/1112319

Change #1112317 merged by jenkins-bot:

[mediawiki/core@REL1_43] dumps: Use proc_close() to close proc_open() subprocess

https://gerrit.wikimedia.org/r/1112317

Maintenance_bot removed a project: Patch-For-Review.Jan 18 2025, 6:30 AM

ReleaseTaggerBot added projects: MW-1.42-notes, MW-1.43-notes, MW-1.39-notes.Jan 18 2025, 7:01 AM

I performed a full dump of eswiki without errors. The file sizes look correct, but I'll double-check by comparing a similar dump from earlier this month at: https://dumps.wikimedia.org/eswiki/latest/

I'm also now dumping enwiki on deployment-snapshot05 so I will see if this also looks good.

My enwiki dumps seems to have stopped making progress at:

2025-01-23 05:09:29: enwiki Checksumming enwiki-20250122-flow.xml.bz2 via sha1
command /usr/bin/python3 xmlflow.py --config /etc/dumps/confs/wikidump.conf.labs --wiki enwiki --outfile /mnt/dumpsdata/xmldatadumps/public/enwiki/20250122/enwiki-20250122-flowhistory.xml.bz2.inprog --history (100664) started...

I checked the data files and they don't seem to have been updated.

btullis@deployment-snapshot05:/mnt/dumpsdata/xmldatadumps/public/enwiki/20250122$ ls -lrt|tail
-rw-r--r-- 1 dumpsgen dumpsgen        63 Jan 23 05:09 md5sums-enwiki-20250122-flow.xml.bz2.txt
-rw-r--r-- 1 dumpsgen dumpsgen        71 Jan 23 05:09 sha1sums-enwiki-20250122-flow.xml.bz2.txt
-rw-r--r-- 1 dumpsgen dumpsgen       460 Jan 23 05:09 status.html
-rw-r--r-- 1 dumpsgen dumpsgen      6879 Jan 23 05:09 report.json
-rw-r--r-- 1 dumpsgen dumpsgen     18697 Jan 23 05:09 index.html
-rw-r--r-- 1 dumpsgen dumpsgen     13159 Jan 23 05:09 dumpstatus.json
-rw-r--r-- 1 dumpsgen dumpsgen       127 Jan 23 05:09 dumpspecialfiles.json
-rw-r--r-- 1 dumpsgen dumpsgen      2832 Jan 23 05:09 dumpruninfo.txt
-rw-r--r-- 1 dumpsgen dumpsgen      3292 Jan 23 05:09 dumpruninfo.json
-rw-r--r-- 1 dumpsgen dumpsgen   2064562 Jan 23 05:27 enwiki-20250122-flowhistory.xml.bz2.inprog
btullis@deployment-snapshot05:/mnt/dumpsdata/xmldatadumps/public/enwiki/20250122$

Investigation continues.

This is what the process tree looks like at the moment, for the dumpsgen user.

btullis@deployment-snapshot05:/mnt/dumpsdata/xmldatadumps/public/enwiki/20250122$ pstree -al dumpsgen
bash
  └─bash ./worker --date 20250122 --onepass --exclusive --log --configfile /etc/dumps/confs/wikidump.conf.labs --wiki enwiki
      └─python3 ./worker.py --configfile /etc/dumps/confs/wikidump.conf.labs --log --exclusive --date 20250122 enwiki
          ├─python3 xmlflow.py --config /etc/dumps/confs/wikidump.conf.labs --wiki enwiki --outfile /mnt/dumpsdata/xmldatadumps/public/enwiki/20250122/enwiki-20250122-flowhistory.xml.bz2.inprog --history
          │   ├─php8.1 /srv/mediawiki/multiversion/MWScript.php extensions/Flow/maintenance/dumpBackup.php --wiki=enwiki --dbgroupdefault=dump --output=file:/mnt/dumpsdata/xmldatadumps/temp/e/enwiki/enwiki-20250122-flowhistory.xml.bz2.inprog_tmp --current --report=1000 --full --skip-header --start=190001 --skip-footer --end 195001
          │   └─sh -c bzip2 >> /mnt/dumpsdata/xmldatadumps/public/enwiki/20250122/enwiki-20250122-flowhistory.xml.bz2.inprog
          │       └─bzip2
          └─3*[{python3}]

If I strace the PHP process, I can see a lot of messages like this:

sendto(7, "add WANCache:global:flow-tree:su"..., 346, MSG_NOSIGNAL, NULL, 0) = 346
sendto(7, "get WANCache:global:flow_revisio"..., 187, MSG_NOSIGNAL, NULL, 0) = 187
recvfrom(7, 0x563639ed8678, 8196, MSG_NOSIGNAL, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

I don't know if this is a true incompatibility between dumps and PHP 8.1, or if it's some kind of misconfiguration related to flow.

I think the reason is that flow has been disabled on enwiki for 8 years:
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/251db04dd46e58c8f76a09b9e90bc0175e5694ff
...but it is still present in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/dblists/flow-labs.dblist#7

I'll make a patch to remove it.

In T382484#10489072, @BTullis wrote:

I think the reason is that flow has been disabled on enwiki for 8 years:
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/251db04dd46e58c8f76a09b9e90bc0175e5694ff
...but it is still present in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/dblists/flow-labs.dblist#7

I'll make a patch to remove it.

Right, came here to say the same, as recent enwiki dumps do not include *flow* files. Too long to list here but I tried:

xcollazo@snapshot1011:/mnt/dumpsdata/xmldatadumps/public$ pwd
/mnt/dumpsdata/xmldatadumps/public
xcollazo@snapshot1011:/mnt/dumpsdata/xmldatadumps/public$ ls -ls ./*/*/*flow*

Only got hits from the other members of that list.

Cool, thanks @xcollazo - For now, I have skipped it by manually removing enwiki from /srv/mediawiki/dblists/flow-labs.dblist and then restarting the job with the --skipdone flag. Now making progress.

2025-01-23 17:10:40: enwiki SUCCESS: done.
Dump of wiki enwiki succeeded.

P72364 T382484 - dump output file listing

1	dumpsgen@deployment-snapshot05:/mnt/dumpsdata/xmldatadumps/public/enwiki$ tree -hD
2	.
3	├── [ 16K Jan 23 17:10] 20250122
4	│ ├── [3.1K Jan 23 17:10] dumpruninfo.json
5	│ ├── [2.7K Jan 23 17:10] dumpruninfo.txt
6	│ ├── [ 127 Jan 23 17:10] dumpspecialfiles.json
7	│ ├── [ 13K Jan 23 17:10] dumpstatus.json
8	│ ├── [2.5M Jan 22 14:40] enwiki-20250122-abstract.xml.gz
9	│ ├── [719K Jan 22 14:24] enwiki-20250122-all-titles-in-ns0.gz
10	│ ├── [2.4M Jan 22 14:24] enwiki-20250122-all-titles.gz
11	│ ├── [ 886 Jan 22 14:24] enwiki-20250122-babel.sql.gz
12	│ ├── [248K Jan 22 14:24] enwiki-20250122-category.sql.gz
13	│ ├── [1.4M Jan 23 15:11] enwiki-20250122-categorylinks.sql.gz
14	│ ├── [1.7M Jan 22 14:24] enwiki-20250122-change_tag.sql.gz
15	│ ├── [1.8K Jan 22 14:24] enwiki-20250122-change_tag_def.sql.gz
16	│ ├── [6.6M Jan 22 14:23] enwiki-20250122-externallinks.sql.gz
17	│ ├── [ 912 Jan 22 14:24] enwiki-20250122-flaggedpage_config.sql.gz
18	│ ├── [ 940 Jan 22 14:24] enwiki-20250122-flaggedpages.sql.gz
19	│ ├── [2.8K Jan 22 14:24] enwiki-20250122-flaggedrevs.sql.gz
20	│ ├── [1.6K Jan 22 14:24] enwiki-20250122-flaggedrevs_promote.sql.gz
21	│ ├── [ 60K Jan 22 14:24] enwiki-20250122-flaggedrevs_statistics.sql.gz
22	│ ├── [1018 Jan 22 14:24] enwiki-20250122-flaggedrevs_tracking.sql.gz
23	│ ├── [ 96M Jan 23 05:09] enwiki-20250122-flow.xml.bz2
24	│ ├── [2.0M Jan 23 05:27] enwiki-20250122-flowhistory.xml.bz2.inprog
25	│ ├── [ 29K Jan 22 14:24] enwiki-20250122-geo_tags.sql.gz
26	│ ├── [ 795 Jan 22 14:24] enwiki-20250122-globalimagelinks.sql.gz
27	│ ├── [170K Jan 23 15:11] enwiki-20250122-image.sql.gz
28	│ ├── [617K Jan 23 15:12] enwiki-20250122-imagelinks.sql.gz
29	│ ├── [110K Jan 22 14:24] enwiki-20250122-iwlinks.sql.gz
30	│ ├── [117K Jan 22 14:23] enwiki-20250122-langlinks.sql.gz
31	│ ├── [4.7M Jan 22 14:23] enwiki-20250122-linktarget.sql.gz
32	│ ├── [3.5K Jan 23 17:10] enwiki-20250122-md5sums.json
33	│ ├── [3.2K Jan 23 17:10] enwiki-20250122-md5sums.txt
34	│ ├── [8.3M Jan 22 14:24] enwiki-20250122-page.sql.gz
35	│ ├── [518K Jan 22 14:24] enwiki-20250122-page_props.sql.gz
36	│ ├── [2.6K Jan 22 14:24] enwiki-20250122-page_restrictions.sql.gz
37	│ ├── [2.6M Jan 23 15:11] enwiki-20250122-pagelinks.sql.gz
38	│ ├── [2.6M Jan 23 17:10] enwiki-20250122-pages-articles-multistream-index.txt.bz2
39	│ ├── [ 65M Jan 23 17:10] enwiki-20250122-pages-articles-multistream.xml.bz2
40	│ ├── [ 62M Jan 22 15:11] enwiki-20250122-pages-articles.xml.bz2
41	│ ├── [ 11M Jan 22 15:44] enwiki-20250122-pages-logging.xml.gz
42	│ ├── [ 67M Jan 22 15:44] enwiki-20250122-pages-meta-current.xml.bz2
43	│ ├── [ 95M Jan 23 17:09] enwiki-20250122-pages-meta-history.xml.7z
44	│ ├── [158M Jan 23 17:08] enwiki-20250122-pages-meta-history.xml.bz2
45	│ ├── [4.1K Jan 22 14:24] enwiki-20250122-protected_titles.sql.gz
46	│ ├── [ 40K Jan 22 14:24] enwiki-20250122-redirect.sql.gz
47	│ ├── [ 63 Jan 22 14:23] enwiki-20250122-runsettings.txt
48	│ ├── [3.8K Jan 23 17:10] enwiki-20250122-sha1sums.json
49	│ ├── [3.6K Jan 23 17:10] enwiki-20250122-sha1sums.txt
50	│ ├── [ 888 Jan 23 15:11] enwiki-20250122-site_stats.sql.gz
51	│ ├── [5.2K Jan 22 14:24] enwiki-20250122-siteinfo-namespaces.json.gz
52	│ ├── [5.4K Jan 22 14:24] enwiki-20250122-siteinfo2-namespacesv2.json.gz
53	│ ├── [2.1K Jan 22 14:24] enwiki-20250122-sites.sql.gz
54	│ ├── [ 16M Jan 22 14:43] enwiki-20250122-stub-articles.xml.gz
55	│ ├── [ 18M Jan 22 14:43] enwiki-20250122-stub-meta-current.xml.gz
56	│ ├── [ 31M Jan 22 14:43] enwiki-20250122-stub-meta-history.xml.gz
57	│ ├── [987K Jan 22 14:23] enwiki-20250122-templatelinks.sql.gz
58	│ ├── [2.9K Jan 22 14:23] enwiki-20250122-user_former_groups.sql.gz
59	│ ├── [3.9K Jan 22 14:23] enwiki-20250122-user_groups.sql.gz
60	│ ├── [5.1K Jan 22 14:24] enwiki-20250122-wbc_entity_usage.sql.gz
61	│ ├── [ 18K Jan 23 17:10] index.html
62	│ ├── [ 66 Jan 22 14:41] md5sums-enwiki-20250122-abstract.xml.gz.txt
63	│ ├── [ 71 Jan 22 14:24] md5sums-enwiki-20250122-all-titles-in-ns0.gz.txt
64	│ ├── [ 64 Jan 22 14:24] md5sums-enwiki-20250122-all-titles.gz.txt
65	│ ├── [ 63 Jan 22 14:24] md5sums-enwiki-20250122-babel.sql.gz.txt
66	│ ├── [ 66 Jan 22 14:24] md5sums-enwiki-20250122-category.sql.gz.txt
67	│ ├── [ 71 Jan 23 15:11] md5sums-enwiki-20250122-categorylinks.sql.gz.txt
68	│ ├── [ 68 Jan 22 14:24] md5sums-enwiki-20250122-change_tag.sql.gz.txt
69	│ ├── [ 72 Jan 22 14:24] md5sums-enwiki-20250122-change_tag_def.sql.gz.txt
70	│ ├── [ 71 Jan 22 14:23] md5sums-enwiki-20250122-externallinks.sql.gz.txt
71	│ ├── [ 76 Jan 22 14:24] md5sums-enwiki-20250122-flaggedpage_config.sql.gz.txt
72	│ ├── [ 70 Jan 22 14:24] md5sums-enwiki-20250122-flaggedpages.sql.gz.txt
73	│ ├── [ 69 Jan 22 14:24] md5sums-enwiki-20250122-flaggedrevs.sql.gz.txt
74	│ ├── [ 77 Jan 22 14:24] md5sums-enwiki-20250122-flaggedrevs_promote.sql.gz.txt
75	│ ├── [ 80 Jan 22 14:24] md5sums-enwiki-20250122-flaggedrevs_statistics.sql.gz.txt
76	│ ├── [ 78 Jan 22 14:24] md5sums-enwiki-20250122-flaggedrevs_tracking.sql.gz.txt
77	│ ├── [ 63 Jan 23 05:09] md5sums-enwiki-20250122-flow.xml.bz2.txt
78	│ ├── [ 66 Jan 22 14:24] md5sums-enwiki-20250122-geo_tags.sql.gz.txt
79	│ ├── [ 74 Jan 22 14:24] md5sums-enwiki-20250122-globalimagelinks.sql.gz.txt
80	│ ├── [ 63 Jan 23 15:11] md5sums-enwiki-20250122-image.sql.gz.txt
81	│ ├── [ 68 Jan 23 15:12] md5sums-enwiki-20250122-imagelinks.sql.gz.txt
82	│ ├── [ 65 Jan 22 14:24] md5sums-enwiki-20250122-iwlinks.sql.gz.txt
83	│ ├── [ 67 Jan 22 14:23] md5sums-enwiki-20250122-langlinks.sql.gz.txt
84	│ ├── [ 68 Jan 22 14:23] md5sums-enwiki-20250122-linktarget.sql.gz.txt
85	│ ├── [ 62 Jan 22 14:24] md5sums-enwiki-20250122-page.sql.gz.txt
86	│ ├── [ 68 Jan 22 14:24] md5sums-enwiki-20250122-page_props.sql.gz.txt
87	│ ├── [ 75 Jan 22 14:24] md5sums-enwiki-20250122-page_restrictions.sql.gz.txt
88	│ ├── [ 67 Jan 23 15:11] md5sums-enwiki-20250122-pagelinks.sql.gz.txt
89	│ ├── [ 91 Jan 23 17:10] md5sums-enwiki-20250122-pages-articles-multistream-index.txt.bz2.txt
90	│ ├── [ 85 Jan 23 17:10] md5sums-enwiki-20250122-pages-articles-multistream.xml.bz2.txt
91	│ ├── [ 73 Jan 22 15:11] md5sums-enwiki-20250122-pages-articles.xml.bz2.txt
92	│ ├── [ 71 Jan 22 15:44] md5sums-enwiki-20250122-pages-logging.xml.gz.txt
93	│ ├── [ 77 Jan 22 15:44] md5sums-enwiki-20250122-pages-meta-current.xml.bz2.txt
94	│ ├── [ 76 Jan 23 17:09] md5sums-enwiki-20250122-pages-meta-history.xml.7z.txt
95	│ ├── [ 77 Jan 23 17:08] md5sums-enwiki-20250122-pages-meta-history.xml.bz2.txt
96	│ ├── [ 74 Jan 22 14:24] md5sums-enwiki-20250122-protected_titles.sql.gz.txt
97	│ ├── [ 66 Jan 22 14:24] md5sums-enwiki-20250122-redirect.sql.gz.txt
98	│ ├── [ 68 Jan 23 15:11] md5sums-enwiki-20250122-site_stats.sql.gz.txt
99	│ ├── [ 78 Jan 22 14:24] md5sums-enwiki-20250122-siteinfo-namespaces.json.gz.txt
100	│ ├── [ 81 Jan 22 14:24] md5sums-enwiki-20250122-siteinfo2-namespacesv2.json.gz.txt
101	│ ├── [ 63 Jan 22 14:24] md5sums-enwiki-20250122-sites.sql.gz.txt
102	│ ├── [ 71 Jan 22 14:43] md5sums-enwiki-20250122-stub-articles.xml.gz.txt
103	│ ├── [ 75 Jan 22 14:43] md5sums-enwiki-20250122-stub-meta-current.xml.gz.txt
104	│ ├── [ 75 Jan 22 14:43] md5sums-enwiki-20250122-stub-meta-history.xml.gz.txt
105	│ ├── [ 71 Jan 22 14:23] md5sums-enwiki-20250122-templatelinks.sql.gz.txt
106	│ ├── [ 76 Jan 22 14:24] md5sums-enwiki-20250122-user_former_groups.sql.gz.txt
107	│ ├── [ 69 Jan 22 14:23] md5sums-enwiki-20250122-user_groups.sql.gz.txt
108	│ ├── [ 74 Jan 22 14:24] md5sums-enwiki-20250122-wbc_entity_usage.sql.gz.txt
109	│ ├── [6.9K Jan 23 17:10] report.json
110	│ ├── [ 74 Jan 22 14:41] sha1sums-enwiki-20250122-abstract.xml.gz.txt
111	│ ├── [ 79 Jan 22 14:24] sha1sums-enwiki-20250122-all-titles-in-ns0.gz.txt
112	│ ├── [ 72 Jan 22 14:24] sha1sums-enwiki-20250122-all-titles.gz.txt
113	│ ├── [ 71 Jan 22 14:24] sha1sums-enwiki-20250122-babel.sql.gz.txt
114	│ ├── [ 74 Jan 22 14:24] sha1sums-enwiki-20250122-category.sql.gz.txt
115	│ ├── [ 79 Jan 23 15:11] sha1sums-enwiki-20250122-categorylinks.sql.gz.txt
116	│ ├── [ 76 Jan 22 14:24] sha1sums-enwiki-20250122-change_tag.sql.gz.txt
117	│ ├── [ 80 Jan 22 14:24] sha1sums-enwiki-20250122-change_tag_def.sql.gz.txt
118	│ ├── [ 79 Jan 22 14:23] sha1sums-enwiki-20250122-externallinks.sql.gz.txt
119	│ ├── [ 84 Jan 22 14:24] sha1sums-enwiki-20250122-flaggedpage_config.sql.gz.txt
120	│ ├── [ 78 Jan 22 14:24] sha1sums-enwiki-20250122-flaggedpages.sql.gz.txt
121	│ ├── [ 77 Jan 22 14:24] sha1sums-enwiki-20250122-flaggedrevs.sql.gz.txt
122	│ ├── [ 85 Jan 22 14:24] sha1sums-enwiki-20250122-flaggedrevs_promote.sql.gz.txt
123	│ ├── [ 88 Jan 22 14:24] sha1sums-enwiki-20250122-flaggedrevs_statistics.sql.gz.txt
124	│ ├── [ 86 Jan 22 14:24] sha1sums-enwiki-20250122-flaggedrevs_tracking.sql.gz.txt
125	│ ├── [ 71 Jan 23 05:09] sha1sums-enwiki-20250122-flow.xml.bz2.txt
126	│ ├── [ 74 Jan 22 14:24] sha1sums-enwiki-20250122-geo_tags.sql.gz.txt
127	│ ├── [ 82 Jan 22 14:24] sha1sums-enwiki-20250122-globalimagelinks.sql.gz.txt
128	│ ├── [ 71 Jan 23 15:11] sha1sums-enwiki-20250122-image.sql.gz.txt
129	│ ├── [ 76 Jan 23 15:12] sha1sums-enwiki-20250122-imagelinks.sql.gz.txt
130	│ ├── [ 73 Jan 22 14:24] sha1sums-enwiki-20250122-iwlinks.sql.gz.txt
131	│ ├── [ 75 Jan 22 14:23] sha1sums-enwiki-20250122-langlinks.sql.gz.txt
132	│ ├── [ 76 Jan 22 14:23] sha1sums-enwiki-20250122-linktarget.sql.gz.txt
133	│ ├── [ 70 Jan 22 14:24] sha1sums-enwiki-20250122-page.sql.gz.txt
134	│ ├── [ 76 Jan 22 14:24] sha1sums-enwiki-20250122-page_props.sql.gz.txt
135	│ ├── [ 83 Jan 22 14:24] sha1sums-enwiki-20250122-page_restrictions.sql.gz.txt
136	│ ├── [ 75 Jan 23 15:11] sha1sums-enwiki-20250122-pagelinks.sql.gz.txt
137	│ ├── [ 99 Jan 23 17:10] sha1sums-enwiki-20250122-pages-articles-multistream-index.txt.bz2.txt
138	│ ├── [ 93 Jan 23 17:10] sha1sums-enwiki-20250122-pages-articles-multistream.xml.bz2.txt
139	│ ├── [ 81 Jan 22 15:11] sha1sums-enwiki-20250122-pages-articles.xml.bz2.txt
140	│ ├── [ 79 Jan 22 15:44] sha1sums-enwiki-20250122-pages-logging.xml.gz.txt
141	│ ├── [ 85 Jan 22 15:44] sha1sums-enwiki-20250122-pages-meta-current.xml.bz2.txt
142	│ ├── [ 84 Jan 23 17:09] sha1sums-enwiki-20250122-pages-meta-history.xml.7z.txt
143	│ ├── [ 85 Jan 23 17:08] sha1sums-enwiki-20250122-pages-meta-history.xml.bz2.txt
144	│ ├── [ 82 Jan 22 14:24] sha1sums-enwiki-20250122-protected_titles.sql.gz.txt
145	│ ├── [ 74 Jan 22 14:24] sha1sums-enwiki-20250122-redirect.sql.gz.txt
146	│ ├── [ 76 Jan 23 15:11] sha1sums-enwiki-20250122-site_stats.sql.gz.txt
147	│ ├── [ 86 Jan 22 14:24] sha1sums-enwiki-20250122-siteinfo-namespaces.json.gz.txt
148	│ ├── [ 89 Jan 22 14:24] sha1sums-enwiki-20250122-siteinfo2-namespacesv2.json.gz.txt
149	│ ├── [ 71 Jan 22 14:24] sha1sums-enwiki-20250122-sites.sql.gz.txt
150	│ ├── [ 79 Jan 22 14:43] sha1sums-enwiki-20250122-stub-articles.xml.gz.txt
151	│ ├── [ 83 Jan 22 14:43] sha1sums-enwiki-20250122-stub-meta-current.xml.gz.txt
152	│ ├── [ 83 Jan 22 14:43] sha1sums-enwiki-20250122-stub-meta-history.xml.gz.txt
153	│ ├── [ 79 Jan 22 14:23] sha1sums-enwiki-20250122-templatelinks.sql.gz.txt
154	│ ├── [ 84 Jan 22 14:24] sha1sums-enwiki-20250122-user_former_groups.sql.gz.txt
155	│ ├── [ 77 Jan 22 14:23] sha1sums-enwiki-20250122-user_groups.sql.gz.txt
156	│ ├── [ 82 Jan 22 14:24] sha1sums-enwiki-20250122-wbc_entity_usage.sql.gz.txt
157	│ ├── [ 107 Jan 23 17:10] status.html
158	│ └── [4.8K Jan 22 14:23] tableinfo.json
159	└── [ 12K Jan 23 17:10] latest
160	├── [ 43 Jan 22 14:41] enwiki-latest-abstract.xml.gz -> ../20250122/enwiki-20250122-abstract.xml.gz
161	├── [ 760 Jan 23 15:12] enwiki-latest-abstract.xml.gz-rss.xml
162	├── [ 48 Jan 22 14:24] enwiki-latest-all-titles-in-ns0.gz -> ../20250122/enwiki-20250122-all-titles-in-ns0.gz
163	├── [ 775 Jan 23 15:12] enwiki-latest-all-titles-in-ns0.gz-rss.xml
164	├── [ 41 Jan 22 14:24] enwiki-latest-all-titles.gz -> ../20250122/enwiki-20250122-all-titles.gz
165	├── [ 754 Jan 23 15:12] enwiki-latest-all-titles.gz-rss.xml
166	├── [ 40 Jan 22 14:24] enwiki-latest-babel.sql.gz -> ../20250122/enwiki-20250122-babel.sql.gz
167	├── [ 751 Jan 23 15:12] enwiki-latest-babel.sql.gz-rss.xml
168	├── [ 43 Jan 22 14:24] enwiki-latest-category.sql.gz -> ../20250122/enwiki-20250122-category.sql.gz
169	├── [ 760 Jan 23 15:12] enwiki-latest-category.sql.gz-rss.xml
170	├── [ 48 Jan 22 14:23] enwiki-latest-categorylinks.sql.gz -> ../20250122/enwiki-20250122-categorylinks.sql.gz
171	├── [ 775 Jan 23 15:12] enwiki-latest-categorylinks.sql.gz-rss.xml
172	├── [ 45 Jan 22 14:24] enwiki-latest-change_tag.sql.gz -> ../20250122/enwiki-20250122-change_tag.sql.gz
173	├── [ 766 Jan 23 15:12] enwiki-latest-change_tag.sql.gz-rss.xml
174	├── [ 49 Jan 22 14:24] enwiki-latest-change_tag_def.sql.gz -> ../20250122/enwiki-20250122-change_tag_def.sql.gz
175	├── [ 778 Jan 23 15:12] enwiki-latest-change_tag_def.sql.gz-rss.xml
176	├── [ 48 Jan 22 14:23] enwiki-latest-externallinks.sql.gz -> ../20250122/enwiki-20250122-externallinks.sql.gz
177	├── [ 775 Jan 23 15:12] enwiki-latest-externallinks.sql.gz-rss.xml
178	├── [ 53 Jan 22 14:24] enwiki-latest-flaggedpage_config.sql.gz -> ../20250122/enwiki-20250122-flaggedpage_config.sql.gz
179	├── [ 790 Jan 23 15:12] enwiki-latest-flaggedpage_config.sql.gz-rss.xml
180	├── [ 47 Jan 22 14:24] enwiki-latest-flaggedpages.sql.gz -> ../20250122/enwiki-20250122-flaggedpages.sql.gz
181	├── [ 772 Jan 23 15:12] enwiki-latest-flaggedpages.sql.gz-rss.xml
182	├── [ 46 Jan 22 14:24] enwiki-latest-flaggedrevs.sql.gz -> ../20250122/enwiki-20250122-flaggedrevs.sql.gz
183	├── [ 769 Jan 23 15:12] enwiki-latest-flaggedrevs.sql.gz-rss.xml
184	├── [ 54 Jan 22 14:24] enwiki-latest-flaggedrevs_promote.sql.gz -> ../20250122/enwiki-20250122-flaggedrevs_promote.sql.gz
185	├── [ 793 Jan 23 15:12] enwiki-latest-flaggedrevs_promote.sql.gz-rss.xml
186	├── [ 57 Jan 22 14:24] enwiki-latest-flaggedrevs_statistics.sql.gz -> ../20250122/enwiki-20250122-flaggedrevs_statistics.sql.gz
187	├── [ 802 Jan 23 15:12] enwiki-latest-flaggedrevs_statistics.sql.gz-rss.xml
188	├── [ 55 Jan 22 14:24] enwiki-latest-flaggedrevs_tracking.sql.gz -> ../20250122/enwiki-20250122-flaggedrevs_tracking.sql.gz
189	├── [ 796 Jan 23 15:12] enwiki-latest-flaggedrevs_tracking.sql.gz-rss.xml
190	├── [ 40 Jan 23 05:09] enwiki-latest-flow.xml.bz2 -> ../20250122/enwiki-20250122-flow.xml.bz2
191	├── [ 751 Jan 23 05:09] enwiki-latest-flow.xml.bz2-rss.xml
192	├── [ 43 Jan 22 14:24] enwiki-latest-geo_tags.sql.gz -> ../20250122/enwiki-20250122-geo_tags.sql.gz
193	├── [ 760 Jan 23 15:12] enwiki-latest-geo_tags.sql.gz-rss.xml
194	├── [ 51 Jan 22 14:24] enwiki-latest-globalimagelinks.sql.gz -> ../20250122/enwiki-20250122-globalimagelinks.sql.gz
195	├── [ 784 Jan 23 15:12] enwiki-latest-globalimagelinks.sql.gz-rss.xml
196	├── [ 40 Jan 22 14:23] enwiki-latest-image.sql.gz -> ../20250122/enwiki-20250122-image.sql.gz
197	├── [ 751 Jan 23 15:12] enwiki-latest-image.sql.gz-rss.xml
198	├── [ 45 Jan 22 14:23] enwiki-latest-imagelinks.sql.gz -> ../20250122/enwiki-20250122-imagelinks.sql.gz
199	├── [ 766 Jan 23 15:12] enwiki-latest-imagelinks.sql.gz-rss.xml
200	├── [ 42 Jan 22 14:24] enwiki-latest-iwlinks.sql.gz -> ../20250122/enwiki-20250122-iwlinks.sql.gz
201	├── [ 757 Jan 23 15:12] enwiki-latest-iwlinks.sql.gz-rss.xml
202	├── [ 44 Jan 22 14:23] enwiki-latest-langlinks.sql.gz -> ../20250122/enwiki-20250122-langlinks.sql.gz
203	├── [ 763 Jan 23 15:12] enwiki-latest-langlinks.sql.gz-rss.xml
204	├── [ 45 Jan 22 14:23] enwiki-latest-linktarget.sql.gz -> ../20250122/enwiki-20250122-linktarget.sql.gz
205	├── [ 766 Jan 23 15:12] enwiki-latest-linktarget.sql.gz-rss.xml
206	├── [ 39 Jan 23 17:10] enwiki-latest-md5sums.txt -> ../20250122/enwiki-20250122-md5sums.txt
207	├── [ 39 Jan 22 14:24] enwiki-latest-page.sql.gz -> ../20250122/enwiki-20250122-page.sql.gz
208	├── [ 748 Jan 23 15:12] enwiki-latest-page.sql.gz-rss.xml
209	├── [ 45 Jan 22 14:24] enwiki-latest-page_props.sql.gz -> ../20250122/enwiki-20250122-page_props.sql.gz
210	├── [ 766 Jan 23 15:12] enwiki-latest-page_props.sql.gz-rss.xml
211	├── [ 52 Jan 22 14:24] enwiki-latest-page_restrictions.sql.gz -> ../20250122/enwiki-20250122-page_restrictions.sql.gz
212	├── [ 787 Jan 23 15:12] enwiki-latest-page_restrictions.sql.gz-rss.xml
213	├── [ 44 Jan 22 14:23] enwiki-latest-pagelinks.sql.gz -> ../20250122/enwiki-20250122-pagelinks.sql.gz
214	├── [ 763 Jan 23 15:12] enwiki-latest-pagelinks.sql.gz-rss.xml
215	├── [ 68 Jan 23 17:10] enwiki-latest-pages-articles-multistream-index.txt.bz2 -> ../20250122/enwiki-20250122-pages-articles-multistream-index.txt.bz2
216	├── [ 835 Jan 23 17:10] enwiki-latest-pages-articles-multistream-index.txt.bz2-rss.xml
217	├── [ 62 Jan 23 17:10] enwiki-latest-pages-articles-multistream.xml.bz2 -> ../20250122/enwiki-20250122-pages-articles-multistream.xml.bz2
218	├── [ 817 Jan 23 17:10] enwiki-latest-pages-articles-multistream.xml.bz2-rss.xml
219	├── [ 50 Jan 22 15:11] enwiki-latest-pages-articles.xml.bz2 -> ../20250122/enwiki-20250122-pages-articles.xml.bz2
220	├── [ 781 Jan 23 15:12] enwiki-latest-pages-articles.xml.bz2-rss.xml
221	├── [ 48 Jan 22 15:44] enwiki-latest-pages-logging.xml.gz -> ../20250122/enwiki-20250122-pages-logging.xml.gz
222	├── [ 775 Jan 23 15:12] enwiki-latest-pages-logging.xml.gz-rss.xml
223	├── [ 54 Jan 22 15:44] enwiki-latest-pages-meta-current.xml.bz2 -> ../20250122/enwiki-20250122-pages-meta-current.xml.bz2
224	├── [ 793 Jan 23 15:12] enwiki-latest-pages-meta-current.xml.bz2-rss.xml
225	├── [ 53 Jan 23 17:09] enwiki-latest-pages-meta-history.xml.7z -> ../20250122/enwiki-20250122-pages-meta-history.xml.7z
226	├── [ 790 Jan 23 17:09] enwiki-latest-pages-meta-history.xml.7z-rss.xml
227	├── [ 54 Jan 23 17:08] enwiki-latest-pages-meta-history.xml.bz2 -> ../20250122/enwiki-20250122-pages-meta-history.xml.bz2
228	├── [ 793 Jan 23 17:08] enwiki-latest-pages-meta-history.xml.bz2-rss.xml
229	├── [ 51 Jan 22 14:24] enwiki-latest-protected_titles.sql.gz -> ../20250122/enwiki-20250122-protected_titles.sql.gz
230	├── [ 784 Jan 23 15:12] enwiki-latest-protected_titles.sql.gz-rss.xml
231	├── [ 43 Jan 22 14:24] enwiki-latest-redirect.sql.gz -> ../20250122/enwiki-20250122-redirect.sql.gz
232	├── [ 760 Jan 23 15:12] enwiki-latest-redirect.sql.gz-rss.xml
233	├── [ 40 Jan 23 17:10] enwiki-latest-sha1sums.txt -> ../20250122/enwiki-20250122-sha1sums.txt
234	├── [ 45 Jan 22 14:23] enwiki-latest-site_stats.sql.gz -> ../20250122/enwiki-20250122-site_stats.sql.gz
235	├── [ 766 Jan 23 15:12] enwiki-latest-site_stats.sql.gz-rss.xml
236	├── [ 55 Jan 22 14:24] enwiki-latest-siteinfo-namespaces.json.gz -> ../20250122/enwiki-20250122-siteinfo-namespaces.json.gz
237	├── [ 796 Jan 23 15:12] enwiki-latest-siteinfo-namespaces.json.gz-rss.xml
238	├── [ 58 Jan 22 14:24] enwiki-latest-siteinfo2-namespacesv2.json.gz -> ../20250122/enwiki-20250122-siteinfo2-namespacesv2.json.gz
239	├── [ 805 Jan 23 15:12] enwiki-latest-siteinfo2-namespacesv2.json.gz-rss.xml
240	├── [ 40 Jan 22 14:24] enwiki-latest-sites.sql.gz -> ../20250122/enwiki-20250122-sites.sql.gz
241	├── [ 751 Jan 23 15:12] enwiki-latest-sites.sql.gz-rss.xml
242	├── [ 48 Jan 22 14:43] enwiki-latest-stub-articles.xml.gz -> ../20250122/enwiki-20250122-stub-articles.xml.gz
243	├── [ 775 Jan 23 15:12] enwiki-latest-stub-articles.xml.gz-rss.xml
244	├── [ 52 Jan 22 14:43] enwiki-latest-stub-meta-current.xml.gz -> ../20250122/enwiki-20250122-stub-meta-current.xml.gz
245	├── [ 787 Jan 23 15:12] enwiki-latest-stub-meta-current.xml.gz-rss.xml
246	├── [ 52 Jan 22 14:43] enwiki-latest-stub-meta-history.xml.gz -> ../20250122/enwiki-20250122-stub-meta-history.xml.gz
247	├── [ 787 Jan 23 15:12] enwiki-latest-stub-meta-history.xml.gz-rss.xml
248	├── [ 48 Jan 22 14:23] enwiki-latest-templatelinks.sql.gz -> ../20250122/enwiki-20250122-templatelinks.sql.gz
249	├── [ 775 Jan 23 15:12] enwiki-latest-templatelinks.sql.gz-rss.xml
250	├── [ 53 Jan 22 14:24] enwiki-latest-user_former_groups.sql.gz -> ../20250122/enwiki-20250122-user_former_groups.sql.gz
251	├── [ 790 Jan 23 15:12] enwiki-latest-user_former_groups.sql.gz-rss.xml
252	├── [ 46 Jan 22 14:23] enwiki-latest-user_groups.sql.gz -> ../20250122/enwiki-20250122-user_groups.sql.gz
253	├── [ 769 Jan 23 15:12] enwiki-latest-user_groups.sql.gz-rss.xml
254	├── [ 51 Jan 22 14:24] enwiki-latest-wbc_entity_usage.sql.gz -> ../20250122/enwiki-20250122-wbc_entity_usage.sql.gz
255	└── [ 784 Jan 23 15:12] enwiki-latest-wbc_entity_usage.sql.gz-rss.xml
256
257	2 directories, 251 files

It's obviously smaller than the real thing, but I imagine that this is just a sensible default for the deployment-prep environment.

I'll double-check this, but it looks promising.

In T382484#10491695, @BTullis wrote:

It's obviously smaller than the real thing, but I imagine that this is just a sensible default for the deployment-prep environment.

Of course! This is the genuine size of https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page

I'm ready to move on to testing the other types of dumps:

In T382484#10491763, @BTullis wrote:

In T382484#10491695, @BTullis wrote:

It's obviously smaller than the real thing, but I imagine that this is just a sensible default for the deployment-prep environment.

Of course! This is the genuine size of https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page

I'm confused, is the database behind https://en.wikipedia.beta.wmflabs.org different than that of the production enwiki?

In T382484#10492468, @xcollazo wrote:

In T382484#10491763, @BTullis wrote:

In T382484#10491695, @BTullis wrote:

It's obviously smaller than the real thing, but I imagine that this is just a sensible default for the deployment-prep environment.

Of course! This is the genuine size of https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page

I'm confused, is the database behind https://en.wikipedia.beta.wmflabs.org different than that of the production enwiki?

Yes, I think so. I see nearly 275k pages here: https://en.wikipedia.beta.wmflabs.org/wiki/Special:Statistics

In T382484#10492468, @xcollazo wrote:

In T382484#10491763, @BTullis wrote:

In T382484#10491695, @BTullis wrote:

It's obviously smaller than the real thing, but I imagine that this is just a sensible default for the deployment-prep environment.

Of course! This is the genuine size of https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page

I'm confused, is the database behind https://en.wikipedia.beta.wmflabs.org different than that of the production enwiki?

Yes, as the banner says:

Hello, and welcome to the WMF integration environment known as Beta Cluster. The software running this wiki is automatically updated whenever there is a change to the software (currently running 1.44.0-alpha (88746d6)). It is used for testing purposes by Wikimedia developers. This wiki is configured to be like en.wikipedia.org

It is configured to be the same, but the data has nothing to do with enwiki.

You can find more information at https://meta.m.wikimedia.beta.wmflabs.org/wiki/Main_Page

The adds/changes dump seems to have worked successfully.

dumpsgen@deployment-snapshot05:/home/btullis$ /usr/bin/python3 /srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py --configfile /etc/dumps/confs/addschanges.conf.labs --dumptype incrdumps
<snip>

[WARNING]: Spawning database subprocess: '/usr/bin/php8.1' '/srv/mediawiki/php-master/../multiversion/MWScript.php' 'fetchText.php' '--wiki' 'zhwikivoyage'
2025-02-11 15:30:29: zhwikivoyage (ID 1449276) 4 pages (9.3|9.3/sec all|curr), 9 revs (20.9|20.9/sec all|curr), ETA 2025-02-11 15:30:31 [max 39]

dumpsgen@deployment-snapshot05:/home/btullis$ echo $?
0

Attempting the wikidata entity dumps now.

dumpsgen@deployment-snapshot05:/home/btullis$ /usr/local/bin/dumpwikibaserdf.sh -c /etc/dumps/confs/wikidump.conf.other.labs -C -p wikidata -d all -f ttl -e nt

It seems to be proceeding with PHP 8.1 without errors, so far.

btullis@deployment-snapshot05:/mnt/dumpsdata/xmldatadumps/temp$ pstree -a dumpsgen
bash
  └─dumpwikibaserdf /usr/local/bin/dumpwikibaserdf.sh -c /etc/dumps/confs/wikidump.conf.other.labs -C -p wikidata -d all -f ttl -e nt
      ├─dumpwikibaserdf /usr/local/bin/dumpwikibaserdf.sh -c /etc/dumps/confs/wikidump.conf.other.labs -C -p wikidata -d all -f ttl -e nt
      │   ├─gzip -9
      │   └─php8.1 /srv/mediawiki/multiversion/MWScript.php extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 0 --sharding-factor 2 --batch-size 500 --format ...
      └─dumpwikibaserdf /usr/local/bin/dumpwikibaserdf.sh -c /etc/dumps/confs/wikidump.conf.other.labs -C -p wikidata -d all -f ttl -e nt
          ├─gzip -9
          └─php8.1 /srv/mediawiki/multiversion/MWScript.php extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 1 --sharding-factor 2 --batch-size 500 --format ...

Looks good.

dumpsgen@deployment-snapshot05:/home/btullis$ /usr/local/bin/dumpwikibaserdf.sh -c /etc/dumps/confs/wikidump.conf.other.labs -C -p wikidata -d all -f ttl -e nt
dumpsgen@deployment-snapshot05:/home/btullis$ echo $?
0

dumpsgen@deployment-snapshot05:/home/btullis$ ls -l /mnt/dumpsdata/otherdumps/wikibase/wikidatawiki/20250211/
total 126032
-rw-r--r-- 1 dumpsgen dumpsgen 32974529 Feb 11 16:15 wikidata-20250211-all-BETA.nt.bz2
-rw-r--r-- 1 dumpsgen dumpsgen 44195982 Feb 11 16:14 wikidata-20250211-all-BETA.nt.gz
-rw-r--r-- 1 dumpsgen dumpsgen 22477641 Feb 11 16:14 wikidata-20250211-all-BETA.ttl.bz2
-rw-r--r-- 1 dumpsgen dumpsgen 29392563 Feb 11 16:14 wikidata-20250211-all-BETA.ttl.gz
-rw-r--r-- 1 dumpsgen dumpsgen      272 Feb 11 16:15 wikidata-20250211-md5sums.txt
-rw-r--r-- 1 dumpsgen dumpsgen      304 Feb 11 16:15 wikidata-20250211-sha1sums.txt

Moving on to categories dumps.

dumpsgen@deployment-snapshot05:/home/btullis$ /usr/local/bin/dumpcategoriesrdf.sh --config /etc/dumps/confs/wikidump.conf.other.labs --list /srv/mediawiki/dblists/categories-rdf.dblist
dumpsgen@deployment-snapshot05:/home/btullis$ echo $?
0

dumpsgen@deployment-snapshot05:/home/btullis$ find /mnt/dumpsdata/otherdumps/categoriesrdf/latest/ -type f|wc -l
758

Attempting a cirrussearch dump of s1.

dumpsgen@deployment-snapshot05:/home/btullis$ /usr/local/bin/dumpcirrussearch.sh --config /etc/dumps/confs/wikidump.conf.other.labs --dblist /srv/mediawiki/dblists/s1.dblist
dumpsgen@deployment-snapshot05:/home/btullis$ echo $?
0

dumpsgen@deployment-snapshot05:/home/btullis$ ls -l /mnt/dumpsdata/otherdumps/cirrussearch/current/
total 134624
-rw-r--r-- 1 dumpsgen dumpsgen 107060425 Feb 11 17:40 enwiki-20250211-cirrussearch-content.json.gz
-rw-r--r-- 1 dumpsgen dumpsgen  30791960 Feb 11 17:42 enwiki-20250211-cirrussearch-general.json.gz

This content translation dump ran very quickly, but looks OK.

dumpsgen@deployment-snapshot05:/home/btullis$ /usr/local/bin/dumpcontentxlation.sh
dumpsgen@deployment-snapshot05:/home/btullis$ echo $?
0
dumpsgen@deployment-snapshot05:/home/btullis$ ls -l /mnt/dumpsdata/otherdumps/contenttranslation/20250211/
total 52
-rw-r--r-- 1 dumpsgen dumpsgen 32986 Feb 11 17:44 cx-corpora._2_.html.json.gz
-rw-r--r-- 1 dumpsgen dumpsgen  4679 Feb 11 17:44 cx-corpora._2_.text.json.gz
-rw-r--r-- 1 dumpsgen dumpsgen  4810 Feb 11 17:44 cx-corpora._2_.text.tmx.gz

The media titles dumps seems to be ok, too: https://wikitech.wikimedia.org/wiki/Dumps/OtherMisc

dumpsgen@deployment-snapshot05:/home/btullis$ /usr/local/bin/create-media-per-project-lists.sh --config /etc/dumps/confs/wikidump.conf.labs:media
dumpsgen@deployment-snapshot05:/home/btullis$ echo $?
0

dumpsgen@deployment-snapshot05:/home/btullis$ tree /mnt/dumpsdata/otherdumps/imageinfo/20250211/|tail -n 1
0 directories, 96 files

Ottomata moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.Feb 11 2025, 6:38 PM

Ottomata added a project: Data-Engineering-Radar.

I think that we can call this done. We have verified that all of the different types of dump now work correctly with PHP 8.1.

🎉 🎉 🎉

Gehel edited projects, added Data-Platform-SRE (2025.02.10 - 2025.02.28); removed Data-Platform-SRE.Feb 24 2025, 7:30 PM

Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2025.02.10 - 2025.02.28) board.Feb 27 2025, 9:29 AM

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.02.10 - 2025.02.28) board.Feb 28 2025, 10:24 AM

Change #1129343 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade snapshot hosts to PHP version 8.1 - except snapshot1016

https://gerrit.wikimedia.org/r/1129343

gerritbot added a project: Patch-For-Review.Mar 19 2025, 6:44 PM

Change #1129343 merged by Btullis:

[operations/puppet@production] Upgrade snapshot hosts to PHP version 8.1 - except snapshot1016

https://gerrit.wikimedia.org/r/1129343

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2025, 7:30 AM

Reedy mentioned this in T389515: PHP Deprecated: Return type of Flow\Search\Iterators\AbstractIterator::current() should either be compatible with Iterator::current(): mixed, or the #[\ReturnTypeWillChange] attribute should be used.Mar 20 2025, 5:31 PM

Change #1132568 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade PHP on the misc dumps worker - snapshot1016

https://gerrit.wikimedia.org/r/1132568

gerritbot added a project: Patch-For-Review.Mar 31 2025, 10:12 AM

Change #1132568 merged by Btullis: