Page MenuHomePhabricator
Create Task
Maniphest T390839

The 20250401 dumps haven't started on time because the mediawikiwiki dump from 20250320 is looping
Closed, ResolvedPublic
Actions

Description

We have an issue whereby the full dumps that were supposed to start on 20250401 at 08:00 have not started.

This seems to be because the mediawikiwiki dump from 20250320 is still running.

We can see that there is an error in the xmlstubsdump job for mediawiki.

2025-04-02 09:32:46: mediawikiwiki (ID 3121830) 12516 pages (3566.0|102390.5/sec all|curr), 25000 revs (7123.0|8180.8/sec all|curr), ETA 2025-04-02 09:50:21 [max 7542525]

Error from command(s): /usr/bin/python3 xmlstubs.py --config /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --articles /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-articles.xml.gz.inprog --history /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-meta-history.xml.gz.inprog --current /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-meta-current.xml.gz.inprog 

2025-04-02 09:41:26: mediawikiwiki *** exception! error producing stub files

2025-04-02 09:41:26: mediawikiwiki ['Traceback (most recent call last):\n', '  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/runner.py", line 454, in do_run_item\n    item.dump(self)\n', '  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/jobs.py", line 183, in dump\n    done = self.run(runner)\n', '  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/xmljobs.py", line 174, in run\n    raise BackupError("error producing stub files")\n', 'dumps.exceptions.BackupError: error producing stub files\n']

Then it seems to say that it will skip the articlesdump job, then it looks like it starts to run.

2025-04-02 09:41:51: mediawikiwiki Required job xmlstubsdump failed, not starting job articlesdump
2025-04-02 09:41:55: mediawikiwiki (ID 2998955) 11 pages (21.2|21.2/sec all|curr), 1000 revs (1925.9|1925.9/sec all|curr), ETA 2025-04-02 10:47:11 [max 7542528]

Meanwhile, all of the other snapshot servers are still looping through the jobs from 20250320

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Fix XmlDumpWriter's handling of unknown content handlersmediawiki/coremaster+17 -3
Temporarily add mediawikiwiki to the skip list for dumpsoperations/puppetproduction+3 -1
Temporarily exclude mediawikiwiki from the dumpsoperations/puppetproduction+13 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Dumps_v1: Re-enable the mediawiki dump again.repos/data-engineering/airflow-dags!1522btullisreenable_mediawikiwiki_dumpmain
test_k8s/dumps/xml_sql: define an explicit skip listrepos/data-engineering/airflow-dags!1472brouberolT390839main
Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
BTullis merged a task: T390605: mediawikiwiki fails some dumps tasks.Apr 2 2025, 10:20 AM
BTullis added a comment.Apr 2 2025, 11:26 AM

Now the mediawikiwiki dump has crashed out on snapshot1012 and has been restarted on snapshot1013

btullis@snapshot1013:~$ sudo lsof -N
COMMAND     PID     USER   FD   TYPE DEVICE  SIZE/OFF       NODE NAME
python3 3138547 dumpsgen    3w   REG   0,54 628995287 2522874146 /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/20250320/dumplog.txt (dumpsdata1006.eqiad.wmnet:/data)
sh      3138695 dumpsgen    1w   REG   0,54 527499862 2522874842 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-meta-history.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
sh      3138696 dumpsgen    1w   REG   0,54 158679638 2522877086 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-meta-current.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
sh      3138697 dumpsgen    1w   REG   0,54 148554326 2522877087 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-articles.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
gzip    3138700 dumpsgen    1w   REG   0,54 527499862 2522874842 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-meta-history.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
gzip    3138701 dumpsgen    1w   REG   0,54 158679638 2522877086 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-meta-current.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
gzip    3138702 dumpsgen    1w   REG   0,54 148554326 2522877087 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250320/mediawikiwiki-20250320-stub-articles.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)

I think that we need to kill all of the remaining process trees for dumpsgen, or somehow convince them that the mediwikiwiki dumps has completed, or failed permanently.

xcollazo added a comment.Apr 2 2025, 1:41 PM

lock suggests process is running on snapshot1012:

xcollazo@snapshot1013:/mnt/dumpsdata/xmldatadumps/private/mediawikiwiki$ cat lock_20250320 
snapshot1012.eqiad.wmnet 119933

But note how the lock file doesn't match the actual run as per dumpadmin.py dryrun:

hostname -f
snapshot1012.eqiad.wmnet

sudo -u dumpsgen bash

cd /srv/deployment/dumps/dumps/xmldumps-backup

python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --dryrun
would kill processes ['120268']

So let's kill, and let's remove lock as well:

python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki
rm /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250320
kill 119933
bash: kill: (119933) - No such process

But it got picked up immediately by snapshot1011, even though the time window to run is already gone:

cat /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250320 
snapshot1011.eqiad.wmnet 3751994
xcollazo added a comment.Apr 2 2025, 2:00 PM

But snapshot1011 shows nothing running? Seems like its dying on its own right away:

hostname -f
snapshot1011.eqiad.wmnet

sudo -u dumpsgen bash

cd /srv/deployment/dumps/dumps/xmldumps-backup

python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --dryrun
would kill processes []

cat /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250320 
snapshot1012.eqiad.wmnet 122938

However, I do see processes on snapshot1011:

$ pstree -a -p dumpsgen | head -n 10
bash,3752685
  |-head,3756933 -n 10
  `-pstree,3756932 -a -p dumpsgen

systemd-timer-m,3576944 /usr/local/bin/systemd-timer-mail-wrapper --subject partialdumps-rest --mail-to ...
  `-fulldumps.sh,3576945 /usr/local/bin/fulldumps.sh 20 25 wikidatawiki partial 28 silent
      `-python3,2161702 /srv/deployment/dumps/dumps/xmldumps-backup/dumpscheduler.py --slots 28 --commands...
          |-sh,2168167 -c...
          |   `-bash,2168168 ./worker --skipdone --exclusive --log --configfile /etc/dumps/confs/wikidump.conf.dumps --date ...
          |       `-python3,3756918 ./worker.py --configfile /etc/dumps/confs/wikidump.conf.dumps --log --job articlesdump ...

Let's just kill it all from the -fulldumps.sh and on:

kill 3576945

pstree -a -p dumpsgen
bash,3752685
  └─pstree,3757069 -a -p dumpsgen

But then it got picked up again:

cat /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250320 
snapshot1012.eqiad.wmnet 122938

rm /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250320 

cat /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250320 
snapshot1013.eqiad.wmnet 3173346
BTullis added a comment.Apr 2 2025, 4:36 PM

We killed all remaining processes that were associated with the 20250320 dump and verified that they had all gone.
The dumps page still shows mediawikiwiki as in progress, but hopefully this will be updated when the 20250401 dump launches.
All being well, that dump should launch this evening. I will monitor it.

image.png (468×851 px, 125 KB)

BTullis added a subscriber: brouberol.Apr 2 2025, 4:42 PM

As far as the jobs failure, I can try running the xmlstubs job manually, to see if it still fails. Here is a dry run.

dumpsgen@snapshot1014:/srv/deployment/dumps/dumps/xmldumps-backup$ python3 ./worker.py --date 20250402 --job xmlstubsdump --configfile /etc/dumps/confs/wikidump.conf.tests --dryrun mediawikiwiki
***
Dry run only, no files will be updated.
***
Running mediawikiwiki, jobs xmlstubsdump...
Couldn't update tableinfo file. Continuing anyways
2025-04-02 16:39:22: mediawikiwiki Cleaning up old dumps for mediawikiwiki
Preparing for job xmlstubsdump of mediawikiwiki
would remove  []
Command to run:  /usr/bin/python3 xmlstubs.py --config /etc/dumps/confs/wikidump.conf.tests --wiki mediawikiwiki --articles /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402/mediawikiwiki-20250402-stub-articles.xml.gz.inprog --history /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402/mediawikiwiki-20250402-stub-meta-history.xml.gz.inprog --current /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402/mediawikiwiki-20250402-stub-meta-current.xml.gz.inprog
2025-04-02 16:39:22: mediawikiwiki Completed job xmlstubsdump for mediawikiwiki

I'll run this in a screen session on snapshot1014 and see if it succeeds.

BTullis added a comment.Apr 2 2025, 4:49 PM

I had to run the createdirs job first.

dumpsgen@snapshot1014:/srv/deployment/dumps/dumps/xmldumps-backup$ python3 ./worker.py --date 20250402 --job createdirs --configfile /etc/dumps/confs/wikidump.conf.tests mediawikiwiki
Running mediawikiwiki, jobs createdirs...
2025-04-02 16:48:11: mediawikiwiki Creating /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402 ...
2025-04-02 16:48:11: mediawikiwiki Checkdir dir /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402 ...
2025-04-02 16:48:11: mediawikiwiki Creating /mnt/dumpsdata/temp/dumpsgen/private/mediawikiwiki/20250402 ...
2025-04-02 16:48:11: mediawikiwiki Cleaning up old dumps for mediawikiwiki
Preparing for job createdirs of mediawikiwiki
2025-04-02 16:48:11: mediawikiwiki Checkdir dir /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402 ...
2025-04-02 16:48:11: mediawikiwiki Checkdir dir /mnt/dumpsdata/temp/dumpsgen/private/mediawikiwiki/20250402 ...
2025-04-02 16:48:12: mediawikiwiki Completed job createdirs for mediawikiwiki

Now I can run the xmlstubsdump command.

dumpsgen@snapshot1014:/srv/deployment/dumps/dumps/xmldumps-backup$ /usr/bin/python3 xmlstubs.py --config /etc/dumps/confs/wikidump.conf.tests --wiki mediawikiwiki --articles /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402/mediawikiwiki-20250402-stub-articles.xml.gz.inprog --history /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402/mediawikiwiki-20250402-stub-meta-history.xml.gz.inprog --current /mnt/dumpsdata/temp/dumpsgen/public/mediawikiwiki/20250402/mediawikiwiki-20250402-stub-meta-current.xml.gz.inprog
BTullis added a comment.Apr 2 2025, 5:24 PM

OK, I see error messages. It looks like it fails and then retries twice.

2025-04-02 17:08:34: mediawikiwiki (ID 1008813) 4110 pages (3342.5|18159.9/sec all|curr), 8000 revs (6506.2|4418.5/sec all|curr), ETA 2025-04-02 17:27:52 [max 7543465]
MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
#1 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
#2 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
#3 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
#4 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 1)
#5 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2125...', false)
#8 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2125...', false)
#9 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2125324, 2145324, false)
#10 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/includes/MaintenanceRunner.php(694): DumpBackup->execute()
#12 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
#14 {main}
nonzero return 1 from command '/usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --report=1000 --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=2125324 --skip-footer --end 2145324'
2025-04-02 17:09:36: mediawikiwiki (ID 1008882) 4110 pages (3543.0|18194.5/sec all|curr), 8000 revs (6896.4|4426.9/sec all|curr), ETA 2025-04-02 17:27:48 [max 7543471]
MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
#1 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
#2 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
#3 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
#4 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 1)
#5 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2125...', false)
#8 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2125...', false)
#9 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2125324, 2145324, false)
#10 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/includes/MaintenanceRunner.php(694): DumpBackup->execute()
#12 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
#14 {main}
nonzero return 1 from command '/usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --report=1000 --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=2125324 --skip-footer --end 2145324'
2025-04-02 17:11:37: mediawikiwiki (ID 1008956) 4110 pages (3401.5|18327.9/sec all|curr), 8000 revs (6621.0|4459.3/sec all|curr), ETA 2025-04-02 17:30:35 [max 7543484]
MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
#1 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
#2 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
#3 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
#4 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 1)
#5 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2125...', false)
#8 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2125...', false)
#9 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2125324, 2145324, false)
#10 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/includes/MaintenanceRunner.php(694): DumpBackup->execute()
#12 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
#14 {main}
nonzero return 1 from command '/usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --report=1000 --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/temp/dumpsgen/temp/m/mediawikiwiki/mediawikiwiki-20250402-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=2125324 --skip-footer --end 2145324'

The last line, immediately after this third failure is:

failed job after max retries
BTullis added a comment.Apr 2 2025, 5:38 PM

I have got a minimal failure case with the following command:

dumpsgen@snapshot1014:/srv/deployment/dumps/dumps/xmldumps-backup$ /usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --filter=latest --filter=notalk --skip-header --start=2125324 --skip-footer --end 2145324 > ~/output.txt

The last entry in the output.txt file is:

<page>
  <title>Help:User page/bn</title>
  <ns>12</ns>
  <id>2131357</id>
  <revision>
    <id>7057255</id>
    <timestamp>2025-02-26T01:03:20Z</timestamp>
    <contributor>
      <username>আফতাবুজ্জামান</username>
      <id>868186</id>
    </contributor>
    <comment>Created page with "সাহায্য:ব্যবহারকারী পাতা"</comment>
    <origin>7057255</origin>
    <model>wikitext</model>
    <format>text/x-wiki</format>
    <text bytes="6224" sha1="lri152h4jxzz8cx7u52dxh4n5ekpxpy" location="es:DB://cluster31/283100?flags=utf-8,gzip" id="44423a2f2f636c757374657233312f323833313030" />
    <sha1>lri152h4jxzz8cx7u52dxh4n5ekpxpy</sha1>
  </revision>
</page>

So I presume that something might be amiss with the next revision in sequence, 2131357

BTullis added a comment.Apr 2 2025, 5:57 PM

Hmm. I am not sure that looking at the next sequence number is really working.
I'm looking at the manual for https://www.mediawiki.org/wiki/Manual:DumpBackup.php and playing around with the --start and --end values, but I can't seem to narrow down exactly what causes it to crash.

BTullis added a comment.Apr 2 2025, 6:15 PM

It is somewhere around here.

If I select the --start value of 2131357 and the --end value of 2131359 then it works.
If I select the --start value of 2131357 and the --end value of 2131360 then it crashes.

dumpsgen@snapshot1014:/srv/deployment/dumps/dumps/xmldumps-backup$ /usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --filter=latest --filter=notalk --skip-header --start=2131357 --skip-footer --end 2131359 > ~/output.txt
2025-04-02 18:10:55: mediawikiwiki (ID 1017867) 1 pages (24.7|24.7/sec all|curr), 1 revs (24.7|24.7/sec all|curr), ETA 2025-04-06 06:50:46 [max 7543580]

dumpsgen@snapshot1014:/srv/deployment/dumps/dumps/xmldumps-backup$ /usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --filter=latest --filter=notalk --skip-header --start=2131357 --skip-footer --end 2131360 > ~/output.txt
MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
#1 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
#2 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
#3 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
#4 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 1)
#5 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2131...', false)
#8 /srv/mediawiki/php-1.44.0-wmf.23/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2131...', false)
#9 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2131357, 2131360, false)
#10 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/includes/MaintenanceRunner.php(694): DumpBackup->execute()
#12 /srv/mediawiki/php-1.44.0-wmf.23/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(219): require_once('/srv/mediawiki/...')
#14 {main}

I've had a look for page ID's with those revision IDs. There are only two.

mysql:research@dbstore1007.eqiad.wmnet [mediawikiwiki]> select * from page where page_id='204918' or page_id='235945';
+---------+----------------+--------------------------------+------------------+-------------+----------------+----------------+--------------------+-------------+----------+--------------------+-----------+
| page_id | page_namespace | page_title                     | page_is_redirect | page_is_new | page_random    | page_touched   | page_links_updated | page_latest | page_len | page_content_model | page_lang |
+---------+----------------+--------------------------------+------------------+-------------+----------------+----------------+--------------------+-------------+----------+--------------------+-----------+
|  204918 |             14 | MediaWiki_components/be        |                0 |           0 |   0.7076217565 | 20250331093010 | 20240124135047     |     4742424 |      373 | wikitext           | NULL      |
|  235945 |             14 | MediaWiki_components/be-tarask |                0 |           0 | 0.255628273122 | 20250331093010 | 20240124135047     |     4742425 |      373 | wikitext           | NULL      |
+---------+----------------+--------------------------------+------------------+-------------+----------------+----------------+--------------------+-------------+----------+--------------------+-----------+
2 rows in set (0.001 sec)

It doesn't mention the content model of JSON here.

I feel that I'm getting out of my depth a little here.

xcollazo added a comment.Apr 2 2025, 7:36 PM

You can find the offending revisions by using the lookup table for conten_models:

mysql:research@dbstore1007.eqiad.wmnet [mediawikiwiki]> select * from content_models;
+----------+-------------------------+
| model_id | model_name              |
+----------+-------------------------+
|       14 | Chart.JsonConfig        |
|        7 | Json.JsonConfig         |  <<<<<<<
|       13 | Map.JsonConfig          |
|        9 | MassMessageListContent  |
|       11 | NewsletterContent       |
|        5 | Scribunto               |
|       12 | Tabular.JsonConfig      |
|        4 | css                     |
|        2 | flow-board              |
|        3 | javascript              |
|        6 | json                    |
|       10 | sanitized-css           |
|        8 | text                    |
|       15 | translate-messagebundle |
|        1 | wikitext                |
+----------+-------------------------+
15 rows in set (0.001 sec)


mysql:research@dbstore1007.eqiad.wmnet [mediawikiwiki]> desc content;
+-----------------+----------------------+------+-----+---------+----------------+
| Field           | Type                 | Null | Key | Default | Extra          |
+-----------------+----------------------+------+-----+---------+----------------+
| content_id      | bigint(20) unsigned  | NO   | PRI | NULL    | auto_increment |
| content_size    | int(10) unsigned     | NO   |     | NULL    |                |
| content_sha1    | varbinary(32)        | NO   |     | NULL    |                |
| content_model   | smallint(5) unsigned | NO   |     | NULL    |                |
| content_address | varbinary(255)       | NO   |     | NULL    |                |
+-----------------+----------------------+------+-----+---------+----------------+
5 rows in set (0.001 sec)

mysql:research@dbstore1007.eqiad.wmnet [mediawikiwiki]> select count(1) from content where content_model = 7;
+----------+
| count(1) |
+----------+
|       14 |
+----------+
1 row in set (3.972 sec)


mysql:research@dbstore1007.eqiad.wmnet [mediawikiwiki]> desc slots;
+------------------+----------------------+------+-----+---------+-------+
| Field            | Type                 | Null | Key | Default | Extra |
+------------------+----------------------+------+-----+---------+-------+
| slot_revision_id | bigint(20) unsigned  | NO   | PRI | NULL    |       |
| slot_role_id     | smallint(5) unsigned | NO   | PRI | NULL    |       |
| slot_content_id  | bigint(20) unsigned  | NO   |     | NULL    |       |
| slot_origin      | bigint(20) unsigned  | NO   |     | NULL    |       |
+------------------+----------------------+------+-----+---------+-------+
4 rows in set (0.002 sec)

select * from slots join content on content.content_id = slots.slot_content_id and content.content_model = 7;
+------------------+--------------+-----------------+-------------+------------+--------------+---------------------------------+---------------+--------------------------------------------+
| slot_revision_id | slot_role_id | slot_content_id | slot_origin | content_id | content_size | content_sha1                    | content_model | content_address                            |
+------------------+--------------+-----------------+-------------+------------+--------------+---------------------------------+---------------+--------------------------------------------+
|          1430977 |            1 |         1212217 |     1430977 |    1212217 |        26249 | qgy46fkpub24ysjs14o3rsl3g5gf44u |             7 | es:DB://cluster24/448691?flags=utf-8,gzip  |
|          1430978 |            1 |         1212218 |     1430978 |    1212218 |        26248 | k64uvyssk7q8i00gio1cr75vyfdmehb |             7 | es:DB://cluster24/448692?flags=utf-8,gzip  |
|          1435870 |            1 |         1216165 |     1435870 |    1216165 |        26248 | k64uvyssk7q8i00gio1cr75vyfdmehb |             7 | es:DB://cluster24/448692?flags=utf-8,gzip  |
|          1517588 |            1 |         1288766 |     1517588 |    1288766 |        26404 | 1kx0upgsg4lcb745o5ligwzegj7gmh7 |             7 | es:DB://cluster24/493086?flags=utf-8,gzip  |
|          1650452 |            1 |         1406573 |     1650452 |    1406573 |         5744 | lup01du1snxg0prrkj9stvbnev6ktga |             7 | es:DB://cluster24/561093?flags=utf-8,gzip  |
|          1650453 |            1 |         1406574 |     1650453 |    1406574 |         5744 | lup01du1snxg0prrkj9stvbnev6ktga |             7 | es:DB://cluster24/561094?flags=utf-8,gzip  |
|          3399898 |            1 |         3288384 |     3399898 |    3288384 |         1430 | m8ocuh94sohai9kcg54n63248pzcqif |             7 | es:DB://cluster25/1634100?flags=utf-8,gzip |
|          3399899 |            1 |         1406574 |     1650453 |    1406574 |         5744 | lup01du1snxg0prrkj9stvbnev6ktga |             7 | es:DB://cluster24/561094?flags=utf-8,gzip  |
|          3660988 |            1 |         3521707 |     3660988 |    3521707 |         1430 | m8ocuh94sohai9kcg54n63248pzcqif |             7 | es:DB://cluster25/1769411?flags=utf-8,gzip |
|          3661089 |            1 |         1406574 |     1650453 |    1406574 |         5744 | lup01du1snxg0prrkj9stvbnev6ktga |             7 | es:DB://cluster24/561094?flags=utf-8,gzip  |
|          3805669 |            1 |         3663921 |     3805669 |    3663921 |         1430 | m8ocuh94sohai9kcg54n63248pzcqif |             7 | es:DB://cluster26/62898?flags=utf-8,gzip   |
|          3805713 |            1 |         3663965 |     3805713 |    3663965 |         1430 | m8ocuh94sohai9kcg54n63248pzcqif |             7 | es:DB://cluster26/62923?flags=utf-8,gzip   |
|          3805714 |            1 |         3663966 |     3805714 |    3663966 |        16165 | 0m26sgtxq5iz1385un9knw1pln4pc0w |             7 | es:DB://cluster26/62924?flags=utf-8,gzip   |
|          6482557 |            1 |         6293647 |     6482557 |    6293647 |           72 | s7fbk7h0th9iza929ardb8i3ngcxvp2 |             7 | es:DB://cluster28/180352?flags=utf-8,gzip  |
|          6482558 |            1 |         6293648 |     6482558 |    6293648 |         8090 | 4ofupjp0a0s7dmxzp7ywe15k7as99cg |             7 | es:DB://cluster29/179851?flags=utf-8,gzip  |
|          7057258 |            1 |         6841917 |     7057258 |    6841917 |        16162 | dcgcwsjqbrzrk7b1ea70tprs5ypy2i3 |             7 | es:DB://cluster30/283325?flags=utf-8,gzip  |
+------------------+--------------+-----------------+-------------+------------+--------------+---------------------------------+---------------+--------------------------------------------+
16 rows in set (15.055 sec)

So for some reason these revisions are failing to find the "content handler", as per the stack on T390839#10705423.

xcollazo added a comment.Apr 3 2025, 1:18 AM

2025-04-01 run of the XML dumps have now started (https://dumps.wikimedia.org/backup-index.html).

However, mediawikiwiki continues with same behavior, so we should solve (or kill again) before the 20th to continue the other dumps.

brouberol mentioned this in T390945: Run an experimental dump of 200 regular sized wikis.Apr 3 2025, 10:24 AM
bking subscribed.Apr 3 2025, 1:38 PM
Poslovitch subscribed.Apr 7 2025, 8:41 AM

Is this why the current dumps are this late? Usually the frwikisource/frwiktionary full dumps are complete in ~3 days from the start of the run, with visible progress happening inbetween. However it seems there have been no progress since April 3rd.

xcollazo added a comment.Apr 7 2025, 1:22 PM
In T390839#10716669, @Poslovitch wrote:

Is this why the current dumps are this late? Usually the frwikisource/frwiktionary full dumps are complete in ~3 days from the start of the run, with visible progress happening inbetween. However it seems there have been no progress since April 3rd.

We suspect that is the case, yes.

I just checked the status, and most wikis have not started the main XML dump. We should kill mediawikiwiki again.

xcollazo added a comment.Apr 7 2025, 1:37 PM
cat /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250401
snapshot1013.eqiad.wmnet 103606

hostname -f
snapshot1013.eqiad.wmnet

sudo -u dumpsgen bash

cd /srv/deployment/dumps/dumps/xmldumps-backup

python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --dryrun
would kill processes ['1036159', '1042016']

python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki
echo $?
0
BTullis added a comment.Apr 7 2025, 10:29 PM

Thanks @xcollazo - Unfortunately, it got picked up again straight away by snapshot1013.

btullis@cumin1002:~$ sudo cumin A:snapshot 'lsof -N|grep mediawikiwiki'
8 hosts will be targeted:
snapshot[1010-1017].eqiad.wmnet
OK to proceed on 8 hosts? Enter the number of affected hosts to confirm or "q" to quit: 8
===== NODE GROUP =====                                                                                                                                                                                             
(1) snapshot1013.eqiad.wmnet                                                                                                                                                                                       
----- OUTPUT of 'lsof -N|grep mediawikiwiki' -----                                                                                                                                                                 
bash    1040870 dumpsgen  cwd    DIR   0,54      4096 2516717447 /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/20250401 (dumpsdata1006.eqiad.wmnet:/data)                                                      
python3 1150526 dumpsgen    3w   REG   0,54 232198205 2516717448 /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/20250401/dumplog.txt (dumpsdata1006.eqiad.wmnet:/data)                                          
python3 1150616 dumpsgen    3r   REG   0,54  12377529 2515142203 /mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250401-stub-articles.xml.gz.inprog_tmp (dumpsdata1006.eqiad.wmnet:/data)
sh      1150653 dumpsgen    1w   REG   0,54 344441430 2522878082 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-meta-history.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
sh      1150654 dumpsgen    1w   REG   0,54  71991894 2522878083 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-meta-current.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
sh      1150655 dumpsgen    1w   REG   0,54  62390870 2522878084 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-articles.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
gzip    1150658 dumpsgen    1w   REG   0,54 344441430 2522878082 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-meta-history.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
gzip    1150659 dumpsgen    1w   REG   0,54  71991894 2522878083 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-meta-current.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
gzip    1150660 dumpsgen    1w   REG   0,54  62407254 2522878084 /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-articles.xml.gz.inprog (dumpsdata1006.eqiad.wmnet:/data)
================                                                                                                                                                                                                   
PASS |████████████████████▋                                                                                                                                                |  12% (1/8) [00:03<00:22,  3.20s/hosts]
FAIL |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                    |  88% (7/8) [00:03<00:00,  2.19hosts/s]
87.5% (7/8) of nodes failed to execute command 'lsof -N|grep mediawikiwiki': snapshot[1010-1012,1014-1017].eqiad.wmnet
12.5% (1/8) success ratio (< 100.0% threshold) for command: 'lsof -N|grep mediawikiwiki'. Aborting.: snapshot1013.eqiad.wmnet
12.5% (1/8) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.: snapshot1013.eqiad.wmnet
btullis@cumin1002:~$

I think that there is a way to cause it to skip a wiki altogether by means of a maintenance.txt file, or similar. I will look it up.

BTullis added a comment.Apr 7 2025, 10:32 PM
In T390839#10719603, @BTullis wrote:

I think that there is a way to cause it to skip a wiki altogether by means of a maintenance.txt file, or similar. I will look it up.

Oh, actually, I think I am mistaken. This is a way of taking a snapshot server out of use, rather than skipping a wiki. From: https://wikitech.wikimedia.org/wiki/Dumps/SQL-XML_Dumps#Deploying_code_updates

Dumps maintenance mode is triggered per worker, by touching the file "/srv/deployment/dumps/dumps/xmldumps-backup/maintenance.txt". This is specific to the version of the dumps scripts running; deployment and running of a new version will bring the worker host out of maintenance mode automatically because the maintenance.txt file will be in the previous version's base directory and not the current version.

xcollazo added a subscriber: Ladsgroup.Apr 8 2025, 1:51 PM

@Ladsgroup:

Pinging you for if you have seen this before. The XML dumps are failing and retrying indefnitely for mediawikiwiki and because of this failure, other wikis cannot progress.

On T390839#10705216, we identified what seems to be the issue:

2025-04-02 17:08:34: mediawikiwiki (ID 1008813) 4110 pages (3342.5|18159.9/sec all|curr), 8000 revs (6506.2|4418.5/sec all|curr), ETA 2025-04-02 17:27:52 [max 7543465]
MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-1.44.0-wmf.23/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
...

We do indeed have a ~16 revisions that use this content model as per DB perusal done on T390839#10705791.

Do you know if there is a way to tell mediawikiwiki to ignore these revisions?

Ladsgroup added a comment.Apr 8 2025, 1:54 PM

There are ways to tackle this. Give me a bit though. Dealing with a couple other major issues right now.

mfossati subscribed.Apr 8 2025, 2:16 PM
Ladsgroup added a comment.Apr 8 2025, 6:21 PM

So the dumper shouldn't try to dump deleted revisions, and most of those revisions were deleted (as admin, it still didn't let me view them, impressive). But these revisions were not deleted:

`
mysql:research@s3-analytics-replica.eqiad.wmnet [mediawikiwiki]> select page_namespace, page_title, rev_id from page join revision on rev_page = page_id where rev_id in (select slot_revision_id from slots join c
ontent on content.content_id = slots.slot_content_id and content.content_model = 7);
+----------------+-------------------+---------+
| page_namespace | page_title        | rev_id  |
+----------------+-------------------+---------+
|              0 | Data:Json:Wikicon | 1430977 |
|              0 | Data:Json:Wikicon | 1430978 |
|              0 | Data:Json:Wikicon | 1435870 |
|              0 | Data:Json:Wikicon | 1517588 |
|              0 | Data:Json:Wikicon | 3805714 |
+----------------+-------------------+---------+

So I marked them for deletion and it should unblock the dumper. Can you try again?

Of course a better solution would be to set Null Content model handler for that (we did for jade in beta cluster) and/or mark those revision as bad in the database but meh life is short.

mfossati mentioned this in T385865: Resume data pipeline operations.Apr 9 2025, 9:13 AM
xcollazo added a comment.Apr 9 2025, 2:32 PM

Thanks @Ladsgroup!

Ran the following:

cat /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250401
snapshot1013.eqiad.wmnet 1657429

hostname -f
snapshot1013.eqiad.wmnet

sudo -u dumpsgen bash

cd /srv/deployment/dumps/dumps/xmldumps-backup

python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --dryrun
would kill processes ['1657541', '1662368']

python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki
echo $?
0

It got immediately picked up again by:

cat /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/lock_20250401
snapshot1010.eqiad.wmnet 1382524

However, I do hope that the new run doesn't run into trouble as per the deleted revisions above.

BTullis added a comment.Apr 10 2025, 11:55 AM

Unfortunately, it look slike it is still looping.
It's currently running on snapshot1012 and this crash happened recently. Same thing as before.

2025-04-10 11:21:56: mediawikiwiki (ID 1638061) 1128 pages (2388.6|10555.4/sec all|curr), 2000 revs (4235.1|9357.6/sec all|curr), ETA 2025-04-10 11:51:41 [max 7559318]
Error from command(s): /usr/bin/python3 xmlstubs.py --config /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --articles /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-
stub-articles.xml.gz.inprog --history /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-meta-history.xml.gz.inprog --current /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/2
0250401/mediawikiwiki-20250401-stub-meta-current.xml.gz.inprog 
2025-04-10 11:31:51: mediawikiwiki *** exception! error producing stub files
2025-04-10 11:31:51: mediawikiwiki ['Traceback (most recent call last):\n', '  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/runner.py", line 454, in
 do_run_item\n    item.dump(self)\n', '  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/jobs.py", line 183, in dump\n    done = self.run(runner)\n', '
  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/xmljobs.py", line 174, in run\n    raise BackupError("error producing stub files")\n', 'dumps.excepti
ons.BackupError: error producing stub files\n']
xcollazo added a comment.Apr 10 2025, 1:55 PM

Ok, a nuclear option here for the time being is to exclude mediawikiwiki from the dumps.

This was done previously for labswiki, see commit.

I'll open up a similar commit for mediawikiwiki while we discuss if this is the best approach for now, but it is the 10th of the month and we got only 3 dumps running due to this issue.

gerritbot added a comment.Apr 10 2025, 2:06 PM

Change #1135734 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/puppet@production] Temporarily exclude mediawikiwiki from the dumps due to multiple failures that don't allow other dumps to move forward.

https://gerrit.wikimedia.org/r/1135734

gerritbot added a project: Patch-For-Review.Apr 10 2025, 2:06 PM
gerritbot added a comment.Apr 11 2025, 9:25 AM

Change #1135734 merged by Btullis:

[operations/puppet@production] Temporarily exclude mediawikiwiki from the dumps

https://gerrit.wikimedia.org/r/1135734

Maintenance_bot removed a project: Patch-For-Review.Apr 11 2025, 9:31 AM
BTullis added a comment.Apr 11 2025, 10:06 AM

I merged that patch and deployed the updated /etc/dumps/dblists/skipmonitor.dblist to all snapshot servers.

Then I killed the running processes on snapshot1010.

dumpsgen@snapshot1010:/srv/deployment/dumps/dumps/xmldumps-backup$ python3 dumpadmin.py --kill --configfile /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --dryrun
would kill processes ['1918724', '1920504']

But it still got picked up again immediately by snapshot1012, so I don't think that the patch worked. I'll have another go at it.

Ladsgroup added a comment.Apr 11 2025, 10:17 AM

The dumper really shouldn't try to load (or worse, try to dump) deleted revisions. If it's doing that, then we have bigger problems :(

gerritbot added a comment.Apr 11 2025, 10:20 AM

Change #1135918 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Temporarily add mediawikiwiki to the skip list for dumps

https://gerrit.wikimedia.org/r/1135918

gerritbot added a project: Patch-For-Review.Apr 11 2025, 10:20 AM
gerritbot added a comment.Apr 11 2025, 10:27 AM

Change #1135918 merged by Btullis:

[operations/puppet@production] Temporarily add mediawikiwiki to the skip list for dumps

https://gerrit.wikimedia.org/r/1135918

Maintenance_bot removed a project: Patch-For-Review.Apr 11 2025, 10:31 AM
BTullis added a comment.Apr 11 2025, 10:58 AM

OK, I think that the patch to skip mediawikiwiki has now worked.
The job still crashed, but this time it did not loop.

The tail of /mnt/dumpsdata/xmldatadumps/private/mediawikiwiki/20250401/dumplog.txt just shows this.

2025-04-11 10:41:56: mediawikiwiki (ID 3046738) 14285 pages (4177.8|125215.8/sec all|curr), 27000 revs (7896.4|8765.5/sec all|curr), ETA 2025-04-11 10:57:50 [max 7560999]
Error from command(s): /usr/bin/python3 xmlstubs.py --config /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --articles /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-articles.xml.gz.inprog --history /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-meta-history.xml.gz.inprog --current /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250401/mediawikiwiki-20250401-stub-meta-current.xml.gz.inprog 
2025-04-11 10:51:58: mediawikiwiki *** exception! error producing stub files
2025-04-11 10:51:58: mediawikiwiki ['Traceback (most recent call last):\n', '  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/runner.py", line 454, in do_run_item\n    item.dump(self)\n', '  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/jobs.py", line 183, in dump\n    done = self.run(runner)\n', '  File "/srv/deployment/dumps/dumps-cache/revs/2fe1059822c2b19208c49406b92c3fc845fc614d/xmldumps-backup/dumps/xmljobs.py", line 174, in run\n    raise BackupError("error producing stub files")\n', 'dumps.exceptions.BackupError: error producing stub files\n']
2025-04-11 10:51:58: mediawikiwiki Required job xmlstubsdump failed, not starting job articlesdump
2025-04-11 10:51:58: mediawikiwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/latest ...
2025-04-11 10:51:59: mediawikiwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/latest ...
2025-04-11 10:52:01: mediawikiwiki Completed job articlesdump for mediawikiwiki

And the job has not been picked up by any other hosts either.

btullis@cumin1002:~$ sudo cumin 'A:snapshot and not D{snapshot1016.eqiad.wmnet}' 'lsof -N|grep mediawikiwiki'
7 hosts will be targeted:
snapshot[1010-1015,1017].eqiad.wmnet
OK to proceed on 7 hosts? Enter the number of affected hosts to confirm or "q" to quit: 7
===== NO OUTPUT =====                                                                                                                                                                                              
PASS |                                                                                                                                                                             |   0% (0/7) [00:02<?, ?hosts/s]
FAIL |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (7/7) [00:02<00:00,  2.53hosts/s]
100.0% (7/7) of nodes failed to execute command 'lsof -N|grep mediawikiwiki': snapshot[1010-1015,1017].eqiad.wmnet
0.0% (0/7) success ratio (< 100.0% threshold) for command: 'lsof -N|grep mediawikiwiki'. Aborting.
0.0% (0/7) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
btullis@cumin1002:~$

I also see lots of article dump jobs running for other wikis now, as well, which will hopefully be reflected on https://dumps.wikimedia.org soon.

BTullis added a comment.Apr 11 2025, 11:16 AM
In T390839#10723140, @Ladsgroup wrote:

So the dumper shouldn't try to dump deleted revisions, and most of those revisions were deleted (as admin, it still didn't let me view them, impressive). But these revisions were not deleted:

`
mysql:research@s3-analytics-replica.eqiad.wmnet [mediawikiwiki]> select page_namespace, page_title, rev_id from page join revision on rev_page = page_id where rev_id in (select slot_revision_id from slots join c
ontent on content.content_id = slots.slot_content_id and content.content_model = 7);
+----------------+-------------------+---------+
| page_namespace | page_title        | rev_id  |
+----------------+-------------------+---------+
|              0 | Data:Json:Wikicon | 1430977 |
|              0 | Data:Json:Wikicon | 1430978 |
|              0 | Data:Json:Wikicon | 1435870 |
|              0 | Data:Json:Wikicon | 1517588 |
|              0 | Data:Json:Wikicon | 3805714 |
+----------------+-------------------+---------+

So I marked them for deletion and it should unblock the dumper. Can you try again?

Of course a better solution would be to set Null Content model handler for that (we did for jade in beta cluster) and/or mark those revision as bad in the database but meh life is short.

I'm sorry to say that I'm struggling to follow this @Ladsgroup
You mention three approaches for potentially dealing with these errors:

  • Mark the revisions for deletion (which you said that you did)
  • Set the Null Content model handler for these revisions
  • Mark the revisions as bad in the database

Personally, I don't know how to do any of these operations and I don't have any wiki admin rights, so I'm not sure how to interpret what you said that you did.

Would you be able to explain how you marked the revisions for deletion please?
Then as a follow-up; how can we check whether or not this has worked?

In T390839#10732835, @Ladsgroup wrote:

The dumper really shouldn't try to load (or worse, try to dump) deleted revisions. If it's doing that, then we have bigger problems :(

Agreed, but once again I don't really know how to look into this further. Thanks for your help.

Ladsgroup added a comment.Apr 11 2025, 11:53 AM
In T390839#10732985, @BTullis wrote:
In T390839#10723140, @Ladsgroup wrote:

So the dumper shouldn't try to dump deleted revisions, and most of those revisions were deleted (as admin, it still didn't let me view them, impressive). But these revisions were not deleted:

`
mysql:research@s3-analytics-replica.eqiad.wmnet [mediawikiwiki]> select page_namespace, page_title, rev_id from page join revision on rev_page = page_id where rev_id in (select slot_revision_id from slots join c
ontent on content.content_id = slots.slot_content_id and content.content_model = 7);
+----------------+-------------------+---------+
| page_namespace | page_title        | rev_id  |
+----------------+-------------------+---------+
|              0 | Data:Json:Wikicon | 1430977 |
|              0 | Data:Json:Wikicon | 1430978 |
|              0 | Data:Json:Wikicon | 1435870 |
|              0 | Data:Json:Wikicon | 1517588 |
|              0 | Data:Json:Wikicon | 3805714 |
+----------------+-------------------+---------+

So I marked them for deletion and it should unblock the dumper. Can you try again?

Of course a better solution would be to set Null Content model handler for that (we did for jade in beta cluster) and/or mark those revision as bad in the database but meh life is short.

I'm sorry to say that I'm struggling to follow this @Ladsgroup
You mention three approaches for potentially dealing with these errors:

  • Mark the revisions for deletion (which you said that you did)
  • Set the Null Content model handler for these revisions

I exampled jade. Here is how it works:

amir@amir:~/mediawiki-config/wmf-config$ grep -ir jade
CommonSettings-labs.php:	// Jade was undeployed as part of T281430, and content is being cleaned up as part of T345874
CommonSettings-labs.php:	$wgContentHandlers['JadeEntity'] = 'FallbackContentHandler';
CommonSettings-labs.php:	$wgContentHandlers['JadeJudgment'] = 'FallbackContentHandler';
  • Mark the revisions as bad in the database

Personally, I don't know how to do any of these operations and I don't have any wiki admin rights, so I'm not sure how to interpret what you said that you did.

Would you be able to explain how you marked the revisions for deletion please?

As an admin, it's an option in the user interface of the wiki to mark a revision for deletion.

Then as a follow-up; how can we check whether or not this has worked?

It didn't work, otherwise it would have not crashed.

In T390839#10732835, @Ladsgroup wrote:

The dumper really shouldn't try to load (or worse, try to dump) deleted revisions. If it's doing that, then we have bigger problems :(

Agreed, but once again I don't really know how to look into this further. Thanks for your help.

I'd say try running the maint script on the specific revisions that are rev deleted and follow the code path. For maint scripts, I don't have a better way to recommend. In user-facing mw side, we have excimer profilers, tracers, xhgui and such but not in maint scripts.

Gehel edited projects, added Data-Platform-SRE (2025.04.12 - 2025.05.02); removed Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:14 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.
Ladsgroup mentioned this in T391858: Masz is down, April 2025.Apr 14 2025, 4:29 PM
xcollazo added a comment.Apr 15 2025, 4:07 PM

An update on XML dumps progress for 2024-01-01 run: all wikis have now finished, except commonswiki and wikidatawiki. Those two are ongoing and are expected to finish successfully.

Thus, the patch to temporarily remove the offending mediawikiwiki worked as expected.

We will leave this wiki on the exclude list until we have time to continue work on this ticket.

xcollazo added a comment.Apr 15 2025, 4:11 PM

Considering @Ladsgroup marked the remaining revisions to not be dumped, I now suspect that the error we saw on T390839#10721567 was a red herring.

We will have to isolate and test mediawikiwiki to properly assess the error.

Ahoelzl added a parent task: T378559: Data Platform Data Incidents.Apr 15 2025, 4:15 PM
jberkel subscribed.Apr 21 2025, 11:44 AM

This also affects the HTML dumps, right? There are none in https://dumps.wikimedia.org/other/enterprise_html/runs/20250401/ or https://dumps.wikimedia.org/other/enterprise_html/runs/20250420/.

xcollazo added a comment.Apr 21 2025, 2:26 PM
In T390839#10756813, @jberkel wrote:

This also affects the HTML dumps, right? There are none in https://dumps.wikimedia.org/other/enterprise_html/runs/20250401/ or https://dumps.wikimedia.org/other/enterprise_html/runs/20250420/.

The HTML dumps have been removed by the Enterprise team (see https://dumps.wikimedia.org/other/enterprise_html/ for alternatives).

We do have a cleanup task to stop producing those empty folders at T390556.

xcollazo added a comment.Apr 21 2025, 2:37 PM
In T390839#10744485, @xcollazo wrote:

An update on XML dumps progress for 2024-01-01 run: all wikis have now finished, except commonswiki and wikidatawiki. Those two are ongoing and are expected to finish successfully.

Thus, the patch to temporarily remove the offending mediawikiwiki worked as expected.

We will leave this wiki on the exclude list until we have time to continue work on this ticket.

Final update on the 2024-01-01 run: All wikis (except mediawikiwiki) finished successfully.

wikidatawiki finished after the 20th of the month, thus required a manual rsync.

jberkel added a comment.Apr 21 2025, 6:56 PM
In T390839#10757036, @xcollazo wrote:
In T390839#10756813, @jberkel wrote:

This also affects the HTML dumps, right? There are none in https://dumps.wikimedia.org/other/enterprise_html/runs/20250401/ or https://dumps.wikimedia.org/other/enterprise_html/runs/20250420/.

The HTML dumps have been removed by the Enterprise team (see https://dumps.wikimedia.org/other/enterprise_html/ for alternatives).

Thanks for the update. Do you know why these have been discontinued? T389542 mentions deprecated endpoints, but couldn't the mirror be updated to use the new v2 endpoints? The dumps are a lot less accessible now (need to have enterprise account, API limits, etc.)

xcollazo added a comment.Apr 21 2025, 8:08 PM
In T390839#10757560, @jberkel wrote:
In T390839#10757036, @xcollazo wrote:
In T390839#10756813, @jberkel wrote:

This also affects the HTML dumps, right? There are none in https://dumps.wikimedia.org/other/enterprise_html/runs/20250401/ or https://dumps.wikimedia.org/other/enterprise_html/runs/20250420/.

The HTML dumps have been removed by the Enterprise team (see https://dumps.wikimedia.org/other/enterprise_html/ for alternatives).

Thanks for the update. Do you know why these have been discontinued? T389542 mentions deprecated endpoints, but couldn't the mirror be updated to use the new v2 endpoints? The dumps are a lot less accessible now (need to have enterprise account, API limits, etc.)

The enterprise HTML dumps code had no maintainer, and it used to fail frequently with no resolution. Now that the endpoints got deprecated, the Enterprise folks decided to just remove it. I agree it is less accessible now, but at least it runs on a supported, reliable mechanism. AFAIU, the API limits should not affect you if you run your workload from WMCS (i.e our free compute platform).

Feel free to bring up your concern over at T389542 though if you feel strongly about it.

jberkel mentioned this in T389542: NEW/CHANGE FEATURE REQUEST: Documentation for v1 Enterprise endpoint deprecation .Apr 22 2025, 12:02 PM
BTullis added a comment.Apr 29 2025, 1:54 PM

I retried my minimal test case above and it continued to fail.

dumpsgen@snapshot1014:/srv/deployment/dumps/dumps/xmldumps-backup$ /usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --filter=latest --filter=notalk --skip-header --start=2125324 --skip-footer --end 2145324 > ~/output.txt
<snip>
2025-04-29 13:46:15: mediawikiwiki (ID 2505892) 56 pages (47.5|47.5/sec all|curr), 100 revs (84.9|84.9/sec all|curr), ETA 2025-04-30 14:38:36 [max 7597693]
<snip>
2025-04-29 13:46:16: mediawikiwiki (ID 2505892) 4398 pages (2048.2|440672.5/sec all|curr), 8600 revs (4005.1|10019.8/sec all|curr), ETA 2025-04-29 14:17:51 [max 7597693]

MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.44.0-wmf.27/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-1.44.0-wmf.27/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
#1 /srv/mediawiki/php-1.44.0-wmf.27/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
#2 /srv/mediawiki/php-1.44.0-wmf.27/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
#3 /srv/mediawiki/php-1.44.0-wmf.27/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
#4 /srv/mediawiki/php-1.44.0-wmf.27/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 2)
#5 /srv/mediawiki/php-1.44.0-wmf.27/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-1.44.0-wmf.27/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-1.44.0-wmf.27/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2125...', false)
#8 /srv/mediawiki/php-1.44.0-wmf.27/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2125...', false)
#9 /srv/mediawiki/php-1.44.0-wmf.27/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2125324, 2145324, false)
#10 /srv/mediawiki/php-1.44.0-wmf.27/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-1.44.0-wmf.27/maintenance/includes/MaintenanceRunner.php(694): DumpBackup->execute()
#12 /srv/mediawiki/php-1.44.0-wmf.27/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(221): require_once('/srv/mediawiki/...')
#14 {main}

I'll keep investigating

Milimetric subscribed.Apr 29 2025, 5:40 PM

This page that failed, Extension:Babel/81/ja, id 2125324, has only one revision, not deleted, but it uses content model id 1 === "wikitext". So, something else is going on here, because that is not the error we're getting, the error says it's trying to look for the Json.JsonConfig content model. Amir tried to fix that by deleting all the revisions that reference content model id 7 === "Json.JsonConfig". That seems to be done well on mediawikiwiki as far as I can tell (the queries Amir shared above look good). So, we gotta find how/why the code is trying to load up something else other than what you're telling it to dump here.

Gehel edited projects, added Data-Platform-SRE (2025.05.02 - 2025.05.23); removed Data-Platform-SRE (2025.04.12 - 2025.05.02).May 5 2025, 12:49 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.
Gehel added a project: Data-Engineering.May 6 2025, 12:30 PM
Ahoelzl moved this task from Incoming (new tickets) to Q4 2025 April 1st - June 30th on the Data-Engineering board.May 6 2025, 5:36 PM
Ahoelzl edited projects, added Data-Engineering (Q4 2025 April 1st - June 30th); removed Data-Engineering.
Gehel edited projects, added Data-Platform-SRE (2025.05.24 - 2025.06.13); removed Data-Platform-SRE (2025.05.02 - 2025.05.23).May 23 2025, 1:08 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.
Ahoelzl subscribed.May 28 2025, 4:21 PM

@Milimetric would you have some time to unblock us here?

Gehel edited projects, added Data-Platform-SRE (2025.06.13 - 2025.07.04); removed Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:49 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.
CodeReviewBot added a project: Patch-For-Review.Jun 26 2025, 12:36 PM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1472

test_k8s/dumps/xml_sql: define an explicit skip list

CodeReviewBot added a comment.Jun 26 2025, 1:02 PM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1472

test_k8s/dumps/xml_sql: define an explicit skip list

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2025, 1:32 PM
BTullis added a comment.Jun 27 2025, 1:03 PM

We are now able to replicate this error in the airflow setup and it is identical to when it is run it on a snapshot server.

btullis@deploy1003:~$ kube-env mediawiki-dumps-legacy-deploy dse-k8s-eqiad

btullis@deploy1003:~$ kubectl exec -it mediawiki-dumps-legacy-toolbox-5fc88c9c76-k7l8c -- bash
Defaulted container "toolbox" out of: toolbox, mediawiki-dumps-legacy-resources-tls-proxy

www-data@mediawiki-dumps-legacy-toolbox-5fc88c9c76-k7l8c:/$ /usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --filter=latest --filter=notalk --skip-header --start=2131357 --skip-footer --end 2131360 > ~/output.txt
MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.45.0-wmf.5/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-1.45.0-wmf.5/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
#1 /srv/mediawiki/php-1.45.0-wmf.5/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
#2 /srv/mediawiki/php-1.45.0-wmf.5/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
#3 /srv/mediawiki/php-1.45.0-wmf.5/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
#4 /srv/mediawiki/php-1.45.0-wmf.5/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 2)
#5 /srv/mediawiki/php-1.45.0-wmf.5/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-1.45.0-wmf.5/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-1.45.0-wmf.5/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2131...', false)
#8 /srv/mediawiki/php-1.45.0-wmf.5/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2131...', false)
#9 /srv/mediawiki/php-1.45.0-wmf.5/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2131357, 2131360, false)
#10 /srv/mediawiki/php-1.45.0-wmf.5/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-1.45.0-wmf.5/maintenance/includes/MaintenanceRunner.php(691): DumpBackup->execute()
#12 /srv/mediawiki/php-1.45.0-wmf.5/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(221): require_once('/srv/mediawiki/...')
#14 {main}
www-data@mediawiki-dumps-legacy-toolbox-5fc88c9c76-k7l8c:/$

We also checked both with and without the --stub argument, but this didn't affect the result.

gerritbot added a comment.Jun 27 2025, 7:05 PM

Change #1164493 had a related patch set uploaded (by Milimetric; author: Milimetric):

[mediawiki/core@master] Fix XmlDumpWriter's handling of deleted revisions

https://gerrit.wikimedia.org/r/1164493

gerritbot added a project: Patch-For-Review.Jun 27 2025, 7:05 PM
Milimetric added a comment.Jun 27 2025, 7:08 PM

I think I found the problem. XmlDumpWriter was just trying to output the model and format as XML from the ContentHandler without checking that the revision was deleted. I think in this case the Json.JsonConfig content handler was unregistered so it was throwing this error. The amazing thing is that we never ran into this problem! I guess maybe this is the first time we unregister a content handler?

gerritbot added a comment.Jul 1 2025, 5:47 PM

Change #1164493 merged by jenkins-bot:

[mediawiki/core@master] Fix XmlDumpWriter's handling of unknown content handlers

https://gerrit.wikimedia.org/r/1164493

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.9; 2025-07-08).Jul 1 2025, 6:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 1 2025, 6:31 PM
CodeReviewBot added a project: Patch-For-Review.Jul 3 2025, 1:38 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1522

Dumps_v1: Re-enable the mediawiki dump again.

CodeReviewBot added a comment.Jul 3 2025, 2:03 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1522

Dumps_v1: Re-enable the mediawiki dump again.

Maintenance_bot removed a project: Patch-For-Review.Jul 3 2025, 2:30 PM
BTullis added a comment.Jul 3 2025, 4:57 PM

Thanks for the patch.
I added mediawikiwiki back to the dumps config on Airflow and tried again, but it failed with the same error.
https://airflow-test-k8s.wikimedia.org/dags/mediawiki_dumps_sql_xml_regular_l_to_m_full/grid?dag_run_id=scheduled__2025-06-01T08%3A00%3A00%2B00%3A00&task_id=mediawikiwiki.dump_xmlstubsdump_full&map_index=-1

It was using an image built today: mediawiki-multiversion-cli:2025-07-03-130458-publish-81

image.png (392×1 px, 175 KB)

[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:28:11: mediawikiwiki (ID 1461) 5319 pages (2343.4|10785.8/sec all|curr), 11000 revs (4846.3|2027.8/sec all|curr), ETA 2025-07-03 16:54:42 [max 7723179]
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #0 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #1 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #2 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #3 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #4 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 2)
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #5 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #6 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #7 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2124...', false)
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #8 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2124...', false)
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #9 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2124119, 2144119, false)
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #10 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #11 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/includes/MaintenanceRunner.php(691): DumpBackup->execute()
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #12 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #13 /srv/mediawiki/multiversion/MWScript.php(221): require_once('/srv/mediawiki/...')
[2025-07-03, 16:28:11 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #14 {main}
[2025-07-03, 16:29:12 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] nonzero return 1 from command '/usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --report=1000 --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=2124119 --skip-footer --end 2144119'
[2025-07-03, 16:29:12 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:12: mediawikiwiki (ID 1465) 73 pages (216.2|216.2/sec all|curr), 1000 revs (2961.7|2961.7/sec all|curr), ETA 2025-07-03 17:12:39 [max 7723181]
[2025-07-03, 16:29:12 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:12: mediawikiwiki (ID 1465) 813 pages (1580.2|4597.1/sec all|curr), 2000 revs (3887.4|5654.5/sec all|curr), ETA 2025-07-03 17:02:18 [max 7723181]
[2025-07-03, 16:29:12 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:12: mediawikiwiki (ID 1465) 1431 pages (2108.9|8721.8/sec all|curr), 3000 revs (4421.1|6094.9/sec all|curr), ETA 2025-07-03 16:58:18 [max 7723181]
[2025-07-03, 16:29:12 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:12: mediawikiwiki (ID 1465) 2140 pages (2511.2|12326.4/sec all|curr), 4000 revs (4693.9|5760.0/sec all|curr), ETA 2025-07-03 16:56:37 [max 7723181]
[2025-07-03, 16:29:12 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:12: mediawikiwiki (ID 1465) 2768 pages (2606.9|13204.8/sec all|curr), 5000 revs (4709.0|4770.5/sec all|curr), ETA 2025-07-03 16:56:31 [max 7723181]
[2025-07-03, 16:29:13 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:12: mediawikiwiki (ID 1465) 3147 pages (2570.1|19348.2/sec all|curr), 6000 revs (4900.2|6148.1/sec all|curr), ETA 2025-07-03 16:55:27 [max 7723181]
[2025-07-03, 16:29:13 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:13: mediawikiwiki (ID 1465) 3591 pages (2589.7|22140.4/sec all|curr), 7000 revs (5048.2|6165.5/sec all|curr), ETA 2025-07-03 16:54:41 [max 7723181]
[2025-07-03, 16:29:13 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:13: mediawikiwiki (ID 1465) 4375 pages (2787.7|23936.5/sec all|curr), 8000 revs (5097.5|5471.2/sec all|curr), ETA 2025-07-03 16:54:26 [max 7723181]
[2025-07-03, 16:29:13 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:13: mediawikiwiki (ID 1465) 4554 pages (2669.2|33303.4/sec all|curr), 9000 revs (5275.0|7313.0/sec all|curr), ETA 2025-07-03 16:53:35 [max 7723181]
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:13: mediawikiwiki (ID 1465) 4977 pages (2686.7|34024.2/sec all|curr), 10000 revs (5398.3|6836.3/sec all|curr), ETA 2025-07-03 16:53:02 [max 7723181]
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:29:14: mediawikiwiki (ID 1465) 5319 pages (2261.0|10637.6/sec all|curr), 11000 revs (4676.0|1999.9/sec all|curr), ETA 2025-07-03 16:56:43 [max 7723181]
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #0 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #1 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #2 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #3 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #4 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 2)
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #5 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #6 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #7 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2124...', false)
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #8 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2124...', false)
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #9 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2124119, 2144119, false)
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #10 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #11 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/includes/MaintenanceRunner.php(691): DumpBackup->execute()
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #12 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #13 /srv/mediawiki/multiversion/MWScript.php(221): require_once('/srv/mediawiki/...')
[2025-07-03, 16:29:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #14 {main}
[2025-07-03, 16:31:14 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] nonzero return 1 from command '/usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --report=1000 --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=2124119 --skip-footer --end 2144119'
[2025-07-03, 16:31:15 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:14: mediawikiwiki (ID 1469) 73 pages (216.1|216.1/sec all|curr), 1000 revs (2960.8|2960.8/sec all|curr), ETA 2025-07-03 17:14:43 [max 7723185]
[2025-07-03, 16:31:15 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:15: mediawikiwiki (ID 1469) 813 pages (1571.3|4524.9/sec all|curr), 2000 revs (3865.4|5565.7/sec all|curr), ETA 2025-07-03 17:04:32 [max 7723185]
[2025-07-03, 16:31:15 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:15: mediawikiwiki (ID 1469) 1431 pages (2083.0|8439.5/sec all|curr), 3000 revs (4367.0|5897.6/sec all|curr), ETA 2025-07-03 17:00:43 [max 7723185]
[2025-07-03, 16:31:15 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:15: mediawikiwiki (ID 1469) 2140 pages (2472.2|11978.9/sec all|curr), 4000 revs (4621.0|5597.6/sec all|curr), ETA 2025-07-03 16:59:05 [max 7723185]
[2025-07-03, 16:31:15 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:15: mediawikiwiki (ID 1469) 2768 pages (2570.3|13100.1/sec all|curr), 5000 revs (4642.9|4732.7/sec all|curr), ETA 2025-07-03 16:58:58 [max 7723185]
[2025-07-03, 16:31:16 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:15: mediawikiwiki (ID 1469) 3147 pages (2537.5|19273.3/sec all|curr), 6000 revs (4837.9|6124.3/sec all|curr), ETA 2025-07-03 16:57:51 [max 7723185]
[2025-07-03, 16:31:16 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:16: mediawikiwiki (ID 1469) 3591 pages (2548.3|21250.4/sec all|curr), 7000 revs (4967.4|5917.7/sec all|curr), ETA 2025-07-03 16:57:09 [max 7723185]
[2025-07-03, 16:31:16 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:16: mediawikiwiki (ID 1469) 4375 pages (2733.1|22837.6/sec all|curr), 8000 revs (4997.6|5220.0/sec all|curr), ETA 2025-07-03 16:56:59 [max 7723185]
[2025-07-03, 16:31:16 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:16: mediawikiwiki (ID 1469) 4554 pages (2616.8|32631.1/sec all|curr), 9000 revs (5171.5|7165.4/sec all|curr), ETA 2025-07-03 16:56:08 [max 7723185]
[2025-07-03, 16:31:16 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:16: mediawikiwiki (ID 1469) 4977 pages (2661.0|38277.8/sec all|curr), 10000 revs (5346.6|7690.9/sec all|curr), ETA 2025-07-03 16:55:19 [max 7723185]
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:31:16: mediawikiwiki (ID 1469) 5319 pages (2254.7|10883.8/sec all|curr), 11000 revs (4662.9|2046.2/sec all|curr), ETA 2025-07-03 16:58:50 [max 7723185]
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] MediaWiki\Exception\MWUnknownContentModelException from line 181 of /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php: The content model 'Json.JsonConfig' is not registered on this wiki.
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #0 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(236): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('Json.JsonConfig', NULL)
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #1 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(170): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('Json.JsonConfig')
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #2 /srv/mediawiki/php-1.45.0-wmf.8/includes/content/ContentHandlerFactory.php(87): MediaWiki\Content\ContentHandlerFactory->createForModelID('Json.JsonConfig')
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #3 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/XmlDumpWriter.php(476): MediaWiki\Content\ContentHandlerFactory->getContentHandler('Json.JsonConfig')
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #4 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/XmlDumpWriter.php(404): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 2)
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #5 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(539): XmlDumpWriter->writeRevision(Object(stdClass), Array)
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #6 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(477): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #7 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(314): WikiExporter->dumpPages('page_id >= 2124...', false)
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #8 /srv/mediawiki/php-1.45.0-wmf.8/includes/export/WikiExporter.php(212): WikiExporter->dumpFrom('page_id >= 2124...', false)
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #9 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/includes/BackupDumper.php(349): WikiExporter->pagesByRange(2124119, 2144119, false)
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #10 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/dumpBackup.php(86): MediaWiki\Maintenance\BackupDumper->dump(1, 1)
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #11 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/includes/MaintenanceRunner.php(691): DumpBackup->execute()
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #12 /srv/mediawiki/php-1.45.0-wmf.8/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #13 /srv/mediawiki/multiversion/MWScript.php(221): require_once('/srv/mediawiki/...')
[2025-07-03, 16:31:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] #14 {main}
[2025-07-03, 16:35:17 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] nonzero return 1 from command '/usr/bin/php8.1 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=mediawikiwiki --dbgroupdefault=dump --full --stub --report=1000 --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/xmldatadumps/temp/m/mediawikiwiki/mediawikiwiki-20250701-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=2124119 --skip-footer --end 2144119'
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] failed job after max retries
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] Error from command(s): /usr/bin/python3 /srv/deployment/dumps/xmldumps-backup/xmlstubs.py --config /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --articles /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250701/mediawikiwiki-20250701-stub-articles.xml.gz.inprog --history /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250701/mediawikiwiki-20250701-stub-meta-history.xml.gz.inprog --current /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250701/mediawikiwiki-20250701-stub-meta-current.xml.gz.inprog
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:35:19: mediawikiwiki *** exception! error producing stub files
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:35:19: mediawikiwiki ['Traceback (most recent call last):\n', '  File "/srv/deployment/dumps/xmldumps-backup/dumps/runner.py", line 454, in do_run_item\n    item.dump(self)\n', '  File "/srv/deployment/dumps/xmldumps-backup/dumps/jobs.py", line 190, in dump\n    done = self.run(runner)\n', '  File "/srv/deployment/dumps/xmldumps-backup/dumps/xmljobs.py", line 178, in run\n    raise BackupError("error producing stub files")\n', 'dumps.exceptions.BackupError: error producing stub files\n']
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:35:19: mediawikiwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/latest ...
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:35:19: mediawikiwiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/latest ...
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-07-03 16:35:19: mediawikiwiki Completed job xmlstubsdump for mediawikiwiki
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] command /usr/bin/python3 /srv/deployment/dumps/xmldumps-backup/xmlstubs.py --config /etc/dumps/confs/wikidump.conf.dumps --wiki mediawikiwiki --articles /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250701/mediawikiwiki-20250701-stub-articles.xml.gz.inprog --history /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250701/mediawikiwiki-20250701-stub-meta-history.xml.gz.inprog --current /mnt/dumpsdata/xmldatadumps/public/mediawikiwiki/20250701/mediawikiwiki-20250701-stub-meta-current.xml.gz.inprog (11) started...
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] returned from 11 with 1
[2025-07-03, 16:35:19 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] returned from 11 with 1
[2025-07-03, 16:35:20 UTC] {pod_manager.py:539} INFO - [mediawiki-dump-sql-xml] Dump of wiki mediawikiwiki failed.

I'll check this out again tomorrow, to make sure that your changes were included in the version of mediawiki deployed. Maybe I need to backport it?

• matmarex subscribed.Jul 3 2025, 8:48 PM

@Milimetric mentioned this on T124748, apparently the error is my fault for incorrectly undeploying some things. We just deployed https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1166236 which should prevent the exception from occurring again.

BTullis closed this task as Resolved.Jul 4 2025, 2:19 PM
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.

@matmarex thanks very much for that - It's now working.
Previoulsy the xmlstubs dump job was failing, but now it has passed that stage and looks like it will complete.

image.png (809×1 px, 140 KB)

BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.Jul 4 2025, 3:57 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits