| xcollazo | |
| Aug 11 2025, 4:27 PM |
| F65740433: image.png | |
| Aug 12 2025, 8:31 AM |
| F65740407: image.png | |
| Aug 12 2025, 8:31 AM |
| F65737399: image.png | |
| Aug 11 2025, 5:07 PM |
| F65737373: image.png | |
| Aug 11 2025, 4:54 PM |
| F65737369: image.png | |
| Aug 11 2025, 4:43 PM |
From XML maling list:
Hello,
I found at least two problems with recent dumps (20250801):
- https://dumps.wikimedia.org/cswiki/20250801/ - some dumps including the most important "pages-articles" failed.
- https://dumps.wikimedia.org/skwiktionary/20250801/ (or without trailing slash) reports 403 Forbidden, same do direct links to dumps inside the dir
Wondering if at least the latter can be fixed soon? Thanks.
Kind regards
Danny B.
Confirmed for cswiki. It failed and we hadn't had a chance to restart it, since Friday.
skwiktionary is a little more nuanced. Here is the DAG Run. (Currently restricted to wmf or nda users, sadly.)
It looks like the initial dump of this had some problems.
I think that I cleared the failed tasks, but I failed to clear a downstream sync task, so the last sync was prior to the successful completion of the dumps.
I checked on the intermediate cephfs volume and the status of the dump is done.
www-data@mediawiki-dumps-legacy-toolbox-5595c8f9dc-8tpng:/mnt/dumpsdata/xmldatadumps/public/skwiktionary/20250801$ cat status.html <li>2025-08-04 21:34:43 <a href="skwiktionary/20250801">skwiktionary</a>: <span class='done'>Dump complete</span></li>
I will clear the sync_remaining_full task, which will hopefully fix this. It appears to be an error in how I reacted to the failed dump, since we know that a sync will always be needed after a re-run.
Oh, this is odd. There seems to be a permissions problem on the file system.
btullis@clouddumps1002:/srv/dumps/xmldatadumps/public/skwiktionary$ cd 20250801/ -bash: cd: 20250801/: Permission denied btullis@clouddumps1002:/srv/dumps/xmldatadumps/public/skwiktionary$ ls -la total 172 drwxrwxr-x 13 dumpsgen dumpsgen 4096 Aug 1 08:17 . drwxr-xr-x 1086 dumpsgen dumpsgen 36864 Aug 11 11:03 .. drwxr-xr-x 2 dumpsgen dumpsgen 12288 May 1 09:42 20250320 drwxr-xr-x 2 dumpsgen dumpsgen 12288 May 20 09:55 20250401 drwxr-xr-x 2 dumpsgen dumpsgen 12288 Jun 1 09:40 20250420 drwxr-xr-x 2 dumpsgen dumpsgen 12288 Jun 20 09:51 20250501 drwxr-xr-x 2 dumpsgen dumpsgen 12288 May 23 10:12 20250520 drwxr-xr-x 2 dumpsgen dumpsgen 12288 Jun 4 10:44 20250601 drwxr-xr-x 2 dumpsgen dumpsgen 12288 Jun 30 15:15 20250620 drwxr-xr-x 2 dumpsgen dumpsgen 12288 Jul 3 05:54 20250701 drwxr-xr-x 2 dumpsgen dumpsgen 12288 Jul 21 04:29 20250720 drwx------ 2 dumpsgen dumpsgen 12288 Aug 4 21:33 20250801 drwxrwxr-x 2 dumpsgen dumpsgen 12288 Aug 5 09:26 latest
I have run chmod 755 on the directory.
btullis@clouddumps1002:/srv/dumps/xmldatadumps/public/skwiktionary$ sudo chmod 755 20250801
Now I can see the page, but the index.html still doesn't seem to be up-to-date.
This suggests that there is something wrong with the sync process.
I can see that no jobs are marked as waiting on the intermediate file store.
www-data@mediawiki-dumps-legacy-toolbox-5595c8f9dc-8tpng:/mnt/dumpsdata/xmldatadumps/public/skwiktionary/20250801$ grep waiting dumpstatus.json www-data@mediawiki-dumps-legacy-toolbox-5595c8f9dc-8tpng:/mnt/dumpsdata/xmldatadumps/public/skwiktionary/20250801$
But on the version that was synced to the clouddumps servers, several jobs are still marked as waiting.
btullis@clouddumps1002:/srv/dumps/xmldatadumps/public/skwiktionary/20250801$ grep waiting dumpstatus.json
{"jobs": {"xmlstubsdump": {"status": "done", "updated": "2025-08-04 13:43:11", "files": {"skwiktionary-20250801-stub-meta-history.xml.gz": {"size": 13116806, "url": "/skwiktionary/20250801/skwiktionary-20250801-stub-meta-history.xml.gz"}, "skwiktionary-20250801-stub-meta-current.xml.gz": {"size": 3268031, "url": "/skwiktionary/20250801/skwiktionary-20250801-stub-meta-current.xml.gz"}, "skwiktionary-20250801-stub-articles.xml.gz": {"size": 3149358, "url": "/skwiktionary/20250801/skwiktionary-20250801-stub-articles.xml.gz"}}}, "xmlpagelogsdump": {"status": "waiting", "updated": "", "files": {"skwiktionary-20250801-pages-logging.xml.gz": {}}}, "wbcentityusagetable": {"status": "done", "updated": "2025-08-02 15:06:20", "files": {"skwiktionary-20250801-wbc_entity_usage.sql.gz": {"size": 16515, "url": "/skwiktionary/20250801/skwiktionary-20250801-wbc_entity_usage.sql.gz", "md5": "c598ed86b1507abe2a1f7618a20054d8", "sha1": "7a486d680008c5053b81da6641d5c1c91640ab1a"}}}, "usergroupstable": {"status": "done", "updated": "2025-08-02 15:05:44", "files": {"skwiktionary-20250801-user_groups.sql.gz": {"size": 910, "url": "/skwiktionary/20250801/skwiktionary-20250801-user_groups.sql.gz", "md5": "5ee6f402bf5336e4333090ac339857cf", "sha1": "d902842da0226d6ed1261fa9b40a07aa6b49d030"}}}, "userformergroupstable": {"status": "done", "updated": "2025-08-02 15:05:47", "files": {"skwiktionary-20250801-user_former_groups.sql.gz": {"size": 763, "url": "/skwiktionary/20250801/skwiktionary-20250801-user_former_groups.sql.gz", "md5": "9779654354445fb7d835e6c09e9716ae", "sha1": "9c17bf0643c5b92810b706a78d51b63d0726ed6c"}}}, "templatelinkstable": {"status": "done", "updated": "2025-08-02 15:05:11", "files": {"skwiktionary-20250801-templatelinks.sql.gz": {"size": 297463, "url": "/skwiktionary/20250801/skwiktionary-20250801-templatelinks.sql.gz", "md5": "6ac86c928e3b0368865d8f330e07b6f1", "sha1": "7ec5632446aa067d83b0bcc25d78cbf41f817833"}}}, "sitestatstable": {"status": "done", "updated": "2025-08-02 15:04:47", "files": {"skwiktionary-20250801-site_stats.sql.gz": {"size": 841, "url": "/skwiktionary/20250801/skwiktionary-20250801-site_stats.sql.gz", "md5": "c2e8332c3dfbfcca84adbe592cfb95e2", "sha1": "567dba7f836530fff02fcac803d4e8f45d20fb92"}}}, "sitestable": {"status": "done", "updated": "2025-08-02 15:06:18", "files": {"skwiktionary-20250801-sites.sql.gz": {"size": 23485, "url": "/skwiktionary/20250801/skwiktionary-20250801-sites.sql.gz", "md5": "8c8b73cd8c14d8441cf83e0321d2eec4", "sha1": "96e43d835ceacf49621f031d28c4206ead6f9dce"}}}, "redirecttable": {"status": "done", "updated": "2025-08-02 15:06:00", "files": {"skwiktionary-20250801-redirect.sql.gz": {"size": 1701, "url": "/skwiktionary/20250801/skwiktionary-20250801-redirect.sql.gz", "md5": "9821988128a9f3e77e59f330669cce69", "sha1": "aeb8411f7a5804fd6de72ecdc40a106e5b76d252"}}}, "protectedtitlestable": {"status": "done", "updated": "2025-08-02 15:05:58", "files": {"skwiktionary-20250801-protected_titles.sql.gz": {"size": 956, "url": "/skwiktionary/20250801/skwiktionary-20250801-protected_titles.sql.gz", "md5": "c603d5c64241d2486ec5a60d707069dc", "sha1": "cfbd351ccabff9e5a99a0204a0761288305314bf"}}}, "pagetitlesdump": {"status": "waiting", "updated": "", "files": {"skwiktionary-20250801-all-titles-in-ns0.gz": {}}}, "pagetable": {"status": "done", "updated": "2025-08-02 15:05:51", "files": {"skwiktionary-20250801-page.sql.gz": {"size": 1199097, "url": "/skwiktionary/20250801/skwiktionary-20250801-page.sql.gz", "md5": "24a4350a6baf6a76cadcaa19d0d4601d", "sha1": "bb0a60079801e478bf8ecf6ef2d9216283bfe012"}}}, "pagerestrictionstable": {"status": "done", "updated": "2025-08-02 15:05:53", "files": {"skwiktionary-20250801-page_restrictions.sql.gz": {"size": 1050, "url": "/skwiktionary/20250801/skwiktionary-20250801-page_restrictions.sql.gz", "md5": "c9db004a9994fd9663a063f26103e4ca", "sha1": "a000da4273c977b6e768eb3dcee08a0e22bcbd4f"}}}, "pagepropstable": {"status": "done", "updated": "2025-08-02 15:05:56", "files": {"skwiktionary-20250801-page_props.sql.gz": {"size": 21588, "url": "/skwiktionary/20250801/skwiktionary-20250801-page_props.sql.gz", "md5": "9d88e31b9f96b062e7418ba0b56d05d2", "sha1": "bd3d812ecbc923899a2ee2a0810f7033c565ef23"}}}, "pagelinkstable": {"status": "done", "updated": "2025-08-02 15:04:58", "files": {"skwiktionary-20250801-pagelinks.sql.gz": {"size": 751074, "url": "/skwiktionary/20250801/skwiktionary-20250801-pagelinks.sql.gz", "md5": "d31993a9ac6000b375b6b7db1612ece7", "sha1": "2454101cfac6317bc7a1890c1b78e6eef31ecffa"}}}, "namespacesv2": {"status": "waiting", "updated": "", "files": {"skwiktionary-20250801-siteinfo2-namespacesv2.json.gz": {}}}, "namespaces": {"status": "done", "updated": "2025-08-04 21:33:32", "files": {"skwiktionary-20250801-siteinfo-namespaces.json.gz": {"size": 6998, "url": "/skwiktionary/20250801/skwiktionary-20250801-siteinfo-namespaces.json.gz"}}}, "metahistorybz2dump": {"status": "waiting", "updated": "", "files": {"skwiktionary-20250801-pages-meta-history.xml.bz2": {}}}, "metahistory7zdump": {"status": "waiting", "updated": ""}, "metacurrentdump": {"status": "done", "updated": "2025-08-04 21:31:25", "files": {"skwiktionary-20250801-pages-meta-current.xml.bz2": {"size": 5899735, "url": "/skwiktionary/20250801/skwiktionary-20250801-pages-meta-current.xml.bz2"}}}, "linktargettable": {"status": "done", "updated": "2025-08-02 15:05:13", "files": {"skwiktionary-20250801-linktarget.sql.gz": {"size": 1427864, "url": "/skwiktionary/20250801/skwiktionary-20250801-linktarget.sql.gz", "md5": "a698467831f1bac472668c25ab1259ea", "sha1": "59309800b954f842fcfadc6d8e737c568e62cbb0"}}}, "langlinkstable": {"status": "done", "updated": "2025-08-02 15:05:42", "files": {"skwiktionary-20250801-langlinks.sql.gz": {"size": 3797171, "url": "/skwiktionary/20250801/skwiktionary-20250801-langlinks.sql.gz", "md5": "1497a02c248609eef64558d363123d46", "sha1": "4e1323dae0f95ea7b81aaba0a3cf37c6ef1bdb7c"}}}, "iwlinkstable": {"status": "done", "updated": "2025-08-02 15:06:02", "files": {"skwiktionary-20250801-iwlinks.sql.gz": {"size": 213914, "url": "/skwiktionary/20250801/skwiktionary-20250801-iwlinks.sql.gz", "md5": "23b6a283e6c6beeb4d1eafe5adb28238", "sha1": "8538aecba6254aa536b9dcf6175162319b0f148a"}}}, "imagetable": {"status": "done", "updated": "2025-08-02 15:04:56", "files": {"skwiktionary-20250801-image.sql.gz": {"size": 1501, "url": "/skwiktionary/20250801/skwiktionary-20250801-image.sql.gz", "md5": "88702b6eaa4d1fbea0cdb20f9d8a229d", "sha1": "4ad33a48f816a3cc248f5bb571698e5d581271e8"}}}, "imagelinkstable": {"status": "done", "updated": "2025-08-02 15:05:08", "files": {"skwiktionary-20250801-imagelinks.sql.gz": {"size": 39590, "url": "/skwiktionary/20250801/skwiktionary-20250801-imagelinks.sql.gz", "md5": "a34424510ee310fc24e3c2ecae97330e", "sha1": "7f12ace37ce3f5a8795e69585aeef060a038bfb0"}}}, "geotagstable": {"status": "done", "updated": "2025-08-02 15:06:04", "files": {"skwiktionary-20250801-geo_tags.sql.gz": {"size": 884, "url": "/skwiktionary/20250801/skwiktionary-20250801-geo_tags.sql.gz", "md5": "c69765ac107648b19747b9589c0ddf39", "sha1": "2d8867bc8e0b5a2a79191a7f69d558f9021a9f30"}}}, "externallinkstable": {"status": "done", "updated": "2025-08-02 15:05:15", "files": {"skwiktionary-20250801-externallinks.sql.gz": {"size": 58564, "url": "/skwiktionary/20250801/skwiktionary-20250801-externallinks.sql.gz", "md5": "a4d4e6051fca8ed5a9d167df4108e183", "sha1": "860526fc5760af6a3039d799afd27994887cf7dd"}}}, "changetagstable": {"status": "done", "updated": "2025-08-02 15:06:06", "files": {"skwiktionary-20250801-change_tag.sql.gz": {"size": 236908, "url": "/skwiktionary/20250801/skwiktionary-20250801-change_tag.sql.gz", "md5": "5d20c4b648be194715bd9ad270e78e30", "sha1": "1260da1664e288a07372a0db88c0f6e671c47e11"}}}, "changetagdeftable": {"status": "done", "updated": "2025-08-02 15:06:08", "files": {"skwiktionary-20250801-change_tag_def.sql.gz": {"size": 1254, "url": "/skwiktionary/20250801/skwiktionary-20250801-change_tag_def.sql.gz", "md5": "dd57ae6da18bd7e585a4ee95874388e1", "sha1": "75ad185642b3aaecab3aea1b303fa56115398bef"}}}, "categorytable": {"status": "done", "updated": "2025-08-02 15:05:49", "files": {"skwiktionary-20250801-category.sql.gz": {"size": 14514, "url": "/skwiktionary/20250801/skwiktionary-20250801-category.sql.gz", "md5": "45242b924a8682f592c552b87115bf23", "sha1": "7353b8285e92e8a883af452bb8da616860dd17ac"}}}, "categorylinkstable": {"status": "done", "updated": "2025-08-02 15:05:05", "files": {"skwiktionary-20250801-categorylinks.sql.gz": {"size": 604134, "url": "/skwiktionary/20250801/skwiktionary-20250801-categorylinks.sql.gz", "md5": "9d6bb6142a1770b5261a31b8f9a7dc8f", "sha1": "f0b1299261f4d0db35a55026f776489c37bcc7c2"}}}, "babeltable": {"status": "done", "updated": "2025-08-02 15:06:27", "files": {"skwiktionary-20250801-babel.sql.gz": {"size": 2211, "url": "/skwiktionary/20250801/skwiktionary-20250801-babel.sql.gz", "md5": "4bdc90fc3f6bf5c8f77062815d0f14d9", "sha1": "2643b9fa13ef761fcd4461f89c09678b6aab8452"}}}, "articlesmultistreamdump": {"status": "done", "updated": "2025-08-04 19:59:28", "files": {"skwiktionary-20250801-pages-articles-multistream.xml.bz2": {"size": 6316127, "url": "/skwiktionary/20250801/skwiktionary-20250801-pages-articles-multistream.xml.bz2"}, "skwiktionary-20250801-pages-articles-multistream-index.txt.bz2": {"size": 266644, "url": "/skwiktionary/20250801/skwiktionary-20250801-pages-articles-multistream-index.txt.bz2"}}}, "articlesdump": {"status": "done", "updated": "2025-08-04 14:21:12", "files": {"skwiktionary-20250801-pages-articles.xml.bz2": {"size": 5477594, "url": "/skwiktionary/20250801/skwiktionary-20250801-pages-articles.xml.bz2"}}}, "allpagetitlesdump": {"status": "waiting", "updated": "", "files": {"skwiktionary-20250801-all-titles.gz": {}}}}, "version": "0.8"}
btullis@clouddumps1002:/srv/dumps/xmldatadumps/public/skwiktionary/20250801$I realise that one of my mistakes was to clear the tasks on skrwiktionary instead of skwiktionary.
Now I have run a sync stage on the correct wiki, the dump files have been published correctly.
The cswiki dump is still problematic, so I will keep looking into this.
It seems to be around here:
[2025-08-11, 16:45:33 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-08-11 16:45:33: cswiki ... building articles 1 XML dump, for output cswiki-20250801-pages-articles1.xml-p1500001p2009898.bz2 with text prefetch from bzip2:/mnt/dumpsdata/xmldatadumps/public/cswiki/20250720/cswiki-20250720-pages-articles1.xml-p1500001p2007940.bz2...
[2025-08-11, 16:45:33 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-08-11 16:45:33: cswiki *** exception! 'NoneType' object has no attribute 'decode'
[2025-08-11, 16:45:33 UTC] {pod_manager.py:520} INFO - [mediawiki-dump-sql-xml] 2025-08-11 16:45:33: cswiki ['Traceback (most recent call last):\n', ' File "/srv/deployment/dumps/xmldumps-backup/dumps/runner.py", line 454, in do_run_item\n item.dump(self)\n', ' File "/srv/deployment/dumps/xmldumps-backup/dumps/jobs.py", line 190, in dump\n done = self.run(runner)\n', ' File "/srv/deployment/dumps/xmldumps-backup/dumps/xmlcontentjobs.py", line 897, in run\n todo = [entry for entry in wanted if not entry[\'generate\'] or\n', ' File "/srv/deployment/dumps/xmldumps-backup/dumps/xmlcontentjobs.py", line 898, in <listcomp>\n not self.stubber.has_no_pages(entry[\'stub\'], runner, tempdir=True)]\n', ' File "/srv/deployment/dumps/xmldumps-backup/dumps/stubprovider.py", line 253, in has_no_pages\n return bool(dcontents.find_first_page_id_in_file() == 0)\n', ' File "/srv/deployment/dumps/xmldumps-backup/dumps/fileutils.py", line 612, in find_first_page_id_in_file\n output = self.get_first_500_lines()\n', ' File "/srv/deployment/dumps/xmldumps-backup/dumps/fileutils.py", line 567, in get_first_500_lines\n return self.first_lines.decode(\'utf-8\')\n', "AttributeError: 'NoneType' object has no attribute 'decode'\n"]The file that was mentioned as the text prefetch checks out as a valid bzip2 file.
www-data@mediawiki-dumps-legacy-toolbox-5595c8f9dc-8tpng:/mnt/dumpsdata/xmldatadumps/public/cswiki/20250720$ bzip2 -t cswiki-20250720-pages-meta-current1.xml-p1500001p2007940.bz2 www-data@mediawiki-dumps-legacy-toolbox-5595c8f9dc-8tpng:/mnt/dumpsdata/xmldatadumps/public/cswiki/20250720$ echo $? 0
I have not been able to rectify this issue with the cswiki dump, so I have a feeling that it might be a problem with the source data.
The partial dumps are about to kick off tomorrow, so it will be interesting to see if this issue affects those dumps as well.
Maybe @xcollazo might have an idea what's going on with this dump, but I won't be able to look into it until the end of the month.
Did trwiktionary dump files also failed? https://dumps.wikimedia.org/trwiktionary/20250801/
Interesting. It shows as success in our internal system. I reran the sync step and it looks like it is good now. Can you please confirm?
Can you please confirm?
Thanks! I downloaded the "pages-articles.xml.bz2" file and it seems to be good.
These are all fixed now. Same fix as in T401610#11106763.
(Side note: I think we need a script to globally rerun rsync on all jobs of a particular run.)
I'll mark this as complete, then create a separate ticket regarding the manual re-sync options and docs.