Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | None | T383053 Container dbs for wikipedia-commons-local-thumb.f8 AWOL in codfw due to corruption | |||
| Resolved | BUG REPORT | MatthewVernon | T384128 wikipedia-commons-local-thumb.4b corrupted causing 401 |
Event Timeline
File:Nintendo-Famicom-Controller-I-FL.jpg has similar issues.
- Go to https://en.wikipedia.org/wiki/Design_around (I'm using Windows 10 and Firefox, both up to date)
- The top image should be a thumbnail of this file, but is replaced with its alt text "Nintendo Famicom controller, with D-pad having a cross shape" linking to the file page
- From the page https://en.wikipedia.org/wiki/Design_around, click on the linked alt text.
- Expected behavior is to display the image full screen, but I see an error message, "Sorry the file cannot be displayed. There seems to be a technical issue. You can retry if it persists. Error could not load image from https://commons.wikimedia.org/wiki/File:Nintendo-Famicom-Controller-I-FL.jpg"
- Go to https://commons.wikimedia.org/wiki/File:Nintendo-Famicom-Controller-I-FL.jpg
- Click on the links following "Other resolutions:"
- Expected behavior is to get different scaled down versions of the image. The full size image (3,600 × 2,520 pixels in this case) loads, but all others give the error "Unauthorized This server could not verify that you are authorized to access the document you requested."
- Go to https://commons.wikimedia.org/wiki/File:Nintendo-Famicom-Controller-I-FL.jpg
- Click "Generate thumbnail" link if you have it enabled
- Expected behavior is to purge the cache. It gives the error message, "This server could not verify that you are authorized to access the document you requested."
Below are archives of both pages. The "Design around" article archives looks exactly how I see the live article. In archives made by both Archive.today and Archive.org, the Commons page displays all images (whereas I see none on the live Commons page).
"Design around" article ( https://en.wikipedia.org/wiki/Design_around ) with broken image: https://archive.today/4jJ8y/image
Commons page (displays correctly in archived version but not the live page??)
https://web.archive.org/web/20250119035348/https://commons.wikimedia.org/wiki/File:Nintendo-Famicom-Controller-I-FL.jpg
http://archive.today/2025.01.19-034537/https://commons.wikimedia.org/wiki/File:Nintendo-Famicom-Controller-I-FL.jpg
For https://commons.wikimedia.org/wiki/File:RotaryIncrementalEncoder.jpg mentioned above, I also cannot see thumbnails on the live page, and I also get the "Unauthorized This server could not verify that you are authorized to access the document you requested." error message when I click the thumbnail links. The full size image link (2,016 × 1,470 pixels) displays as expected: https://upload.wikimedia.org/wikipedia/commons/4/4b/RotaryIncrementalEncoder.jpg I took a archive/screenshot of the page on Archive.today. The archive looks the same as live page (no thumbnails displayed).
Can confirm the codfw db for this container is missing, and there are quarantined versions.
Of interest/concern is that ms-be2068 has quarantined this before:
mvernon@ms-be2068:~$ sudo ls -l /srv/swift-storage/sdb3/quarantined/containers total 0 drwxr-xr-x 2 swift swift 113 Oct 9 2023 451cd039c82e3456c69a99f613023ea3 drwxr-xr-x 2 swift swift 113 Jan 17 20:15 451cd039c82e3456c69a99f613023ea3-5a0f72d826d64ce5a11ea8c729c6e586
The error from the sqlite integrity check for each copy was wrong # of entries in index ix_object_deleted_name, so I reindexed that index and then the integrity check was OK. At which point, I put the quarantined DB back into place, and we seem to be back in business.
It's obviously concerning that this is the second one of these we've seen in quick succession :(
Logs:
Jan 17 20:15:29 ms-be2081 container-server: Quarantined /srv/swift-storage/accounts1/containers/17692/ea3/451cd039c82e3456c69a99f613023ea3 to /srv/swift-storage/accounts1/quarantined/containers/451cd039c82e3456c69a99f613023ea3 due to malformed database (txn: tx3052590571b44296a9379-00678aba61) Jan 17 20:15:29 ms-be2068 container-server: Quarantined /srv/swift-storage/sdb3/containers/17692/ea3/451cd039c82e3456c69a99f613023ea3 to /srv/swift-storage/sdb3/quarantined/containers/451cd039c82e3456c69a99f613023ea3-5a0f72d826d64ce5a11ea8c729c6e586 due to malformed database (txn: tx3052590571b44296a9379-00678aba61) Jan 17 20:15:29 ms-be2071 container-server: Quarantined /srv/swift-storage/accounts0/containers/17692/ea3/451cd039c82e3456c69a99f613023ea3 to /srv/swift-storage/accounts0/quarantined/containers/451cd039c82e3456c69a99f613023ea3 due to malformed database (txn: tx3052590571b44296a9379-00678aba61)
Previous requests for that container (1 per container server):
Jan 17 20:15:19 ms-be2081 container-server: 10.192.16.248 - - [17/Jan/2025:20:15:19 +0000] "PUT /accounts1/17692/AUTH_mw/wikipedia-commons-local-thumb.4b/4/4b/Population_-_Municipality_code_58041.svg/300px-Population_-_Municipality_code_58041.svg.png" 201 - "PUT http://ms-fe.svc.codfw.wmnet/objects2/61202/AUTH_mw/wikipedia-commons-local-thumb.4b/4/4b/Population_-_Municipality_code_58041.svg/300px-Population_-_Municipality_code_58041.svg.png" "tx9a96a50542b147f0b227b-00678aba57" "object-server 2179" 0.0018 "-" 1988 0 Jan 17 20:15:19 ms-be2068 container-server: 10.192.0.86 - - [17/Jan/2025:20:15:19 +0000] "PUT /sdb3/17692/AUTH_mw/wikipedia-commons-local-thumb.4b/4/4b/Population_-_Municipality_code_58041.svg/300px-Population_-_Municipality_code_58041.svg.png" 201 - "PUT http://ms-fe.svc.codfw.wmnet/objects18/61202/AUTH_mw/wikipedia-commons-local-thumb.4b/4/4b/Population_-_Municipality_code_58041.svg/300px-Population_-_Municipality_code_58041.svg.png" "tx9a96a50542b147f0b227b-00678aba57" "object-server 1860" 0.0014 "-" 1766 0 Jan 17 20:15:28 ms-be2071 object-server: 10.192.16.76 - - [17/Jan/2025:20:15:28 +0000] "GET /objects10/14340/AUTH_mw/wikipedia-commons-local-thumb.4b/4/4b/Cervus_elaphus_Luc_Viatour_6.jpg/120px-Cervus_elaphus_Luc_Viatour_6.jpg" 304 - "GET http://127.0.0.1/v1/AUTH_mw/wikipedia-commons-local-thumb.4b/4/4b/Cervus_elaphus_Luc_Viatour_6.jpg/120px-Cervus_elaphus_Luc_Viatour_6.jpg" "tx791cc4796ade4581a6c26-00678aba60" "proxy-server 2679741" 0.0266 "-" 1808 0
The triggering query, via transaction id:
Jan 17 20:15:29 ms-fe2014 proxy-server: 10.194.177.233 10.192.16.194 17/Jan/2025/20/15/29 GET /v1/AUTH_mw/wikipedia-commons-local-thumb.4b%3Flimit%3D9000%26prefix%3D4%252F4b%252FParc_de_l%2527Aigua_-_Daniel_Navas%252C_Neus_Sol%25C3%25A9_Arq._6.jpg%252F%26format%3Djson%26states%3Dlisting HTTP/1.0 503 - wikimedia/multi-http-client%20v1.1 AUTH_tk8644ffd2c... - 118 - tx3052590571b44296a9379-00678aba61 - 0.0933 - - 1737144929.548093081 1737144929.641421080 -
Final successful listing (1 per server):
Jan 17 20:13:09 ms-be2068 container-server: 10.192.16.76 - - [17/Jan/2025:20:13:09 +0000] "HEAD /sdb3/17692/AUTH_mw/wikipedia-commons-local-thumb.4b" 204 - "HEAD http://127.0.0.1/v1/AUTH_mw/wikipedia-commons-local-thumb.4b?format=json&states=listing" "tx3f30f5e14c6f423dae97d-00678ab9d5" "proxy-server 2679729" 0.0098 "-" 1787 0`` Jan 17 20:14:09 ms-be2081 container-server: 10.192.48.44 - - [17/Jan/2025:20:14:09 +0000] "HEAD /accounts1/17692/AUTH_mw/wikipedia-commons-local-thumb.4b" 204 - "HEAD http://127.0.0.1/v1/AUTH_mw/wikipedia-commons-local-thumb.4b?format=json&states=listing" "txa8becbc4388a40e0837c4-00678aba11" "proxy-server 1601011" 0.0039 "-" 1980 0 Jan 17 20:15:09 ms-be2071 container-server: 10.192.0.87 - - [17/Jan/2025:20:15:09 +0000] "HEAD /accounts0/17692/AUTH_mw/wikipedia-commons-local-thumb.4b" 204 - "HEAD http://127.0.0.1/v1/AUTH_mw/wikipedia-commons-local-thumb.4b?format=json&states=listing" "txf52020832d92435dbc2d6-00678aba4d" "proxy-server 61587" 0.0090 "-" 2515 0
Final row in each db (varying only in rowid):
19828769|4/4b/Population_-_Municipality_code_58041.svg/300px-Population_-_Municipality_code_58041.svg.png|1737144919.73295|7229|image/png|938e4e2c4a424339f34248a54e44cefe|0|0
That timestamp is 2025-01-17 20:15:19