| Yann | |
| Thu, May 2, 11:46 AM |
| F50516465: Gnome-edit-delete.svg | |
| Mon, May 6, 1:15 PM |
| F50516466: 20120911151633!Gnome-edit-delete.svg | |
| Mon, May 6, 1:15 PM |
Steps to replicate the issue (include links if applicable):
Tested on Windows 10 with Chrome and Firefox, logged and not logged.
The earliest request in our logs for an original is from 28 Apr, already 404:
moss-fe1001.eqiad.wmnet: /var/log/swift/proxy-access.log.4.gz:Apr 28 10:22:47 moss-fe1001 proxy-server: ELIDED 10.64.0.45 28/Apr/2024/10/22/47 GET /v1/AUTH_mw/wikipedia-commons-local-public.96/9/96/Gnome-edit-delete.svg HTTP/1.0 404 https://commons.wikimedia.org/ Mozilla/5.0%20%28Macintosh%3B%20Intel%20Mac%20OS%20X%2010_15_7%29%20AppleWebKit/605.1.15%20%28KHTML%2C%20like%20Gecko%29%20Version/15.6.1%20Safari/605.1.15 - - 88 - tx0c8d529d70854f0d903d7-00662e2377 - 0.0465 - - 1714299767.782626390 1714299767.829124689 0
There are no DELETEs for the original in swift logs, but there is a whole batch (220) of DELETEs for the thumbnails late on May 1 all within a second or so, e.g.:
moss-fe1001.eqiad.wmnet: /var/log/swift/proxy-access.log.1.gz:May 1 23:29:13 moss-fe1001 proxy-server: 10.64.16.161 10.64.0.45 01/May/2024/23/29/13 DELETE /v1/AUTH_mw/wikipedia-commons-local-thumb.96/9/96/Gnome-edit-delete.svg/21px-Gnome-edit-delete.svg.png HTTP/1.0 204 - wikimedia/multi-http-client%20v1.1 AUTH_tk998b7465d... - - - tx8a5c386d35bc480cb99aa-006632d049 - 0.0757 - - 1714606153.905497789 1714606153.981242895 0
Neither ms swift cluster has the original (per swift stat wikipedia-commons-local-public.96 9/96/Gnome-edit-delete.svg).
The codfw backups know nothing of this file:
mvernon@ms-backup2001:~$ sudo -u mediabackup query-media-file [2024-05-02 13:28:02,405] INFO:query Starting an interactive query session Wiki for query [default: commonswiki]: 0) Title of the file on upload (or after rename) 1) sha1sum hash of the file contents, in hexadecimal 2) sha1sum hash of the file contents, in Mediawiki's base 36 3) Original container name and full path as was stored on Swift 4) sha256sum hash of the file contents, in hexadecimal 5) Exact date of the original file upload, as registered on the metadata 6) Exact date of the latest file archival, as registered on the metadata 7) Exact date of the latest file deletion, as registered on the metadata Chose method to identify the media file to query (0-7) [default: 0]: Title (spaces will be converted to underscores, first letter normally in uppercase): Gnome-edit-delete.svg [2024-05-02 13:28:19,067] WARNING:query No file was found that matched the given criteria, exiting.
But eqiad backups do know about both versions:
mvernon@ms-backup1001:~$ sudo -u mediabackup query-media-file [2024-05-02 13:21:48,417] INFO:query Starting an interactive query session Wiki for query [default: commonswiki]: 0) Title of the file on upload (or after rename) 1) sha1sum hash of the file contents, in hexadecimal 2) sha1sum hash of the file contents, in Mediawiki's base 36 3) Original container name and full path as was stored on Swift 4) sha256sum hash of the file contents, in hexadecimal 5) Exact date of the original file upload, as registered on the metadata 6) Exact date of the latest file archival, as registered on the metadata 7) Exact date of the latest file deletion, as registered on the metadata Chose method to identify the media file to query (0-7) [default: 0]: 0 Title (spaces will be converted to underscores, first letter normally in uppercase): Gnome-edit-delete.svg This is the list of 2 files found with the given criteria: 0) wiki | commonswiki title | Gnome-edit-delete.svg production_container | wikipedia-commons-local-public.96 production_path | 9/96/Gnome-edit-delete.svg sha1 | 3a36632c6569fcdf45aca81a71b53ae4faf80083 sha256 | a45ec2020e0997a031bdd62a0dc30a518c82d9c1a100d6e8420bb2a6f938c48f size | 20144 type | DRAWING production_status | public production_url | https://upload.wikimedia.org/wikipedia/commons/9/96/Gnome-edit-delete.svg upload_date | 2012-09-11 15:16:34 archive_date | None delete_date | None backup_status | backedup backup_date | 2021-09-08 21:45:12 backup_location | https://backup1006.eqiad.wmnet:9000 backup_container | mediabackups backup_path | commonswiki/a45/a45ec2020e0997a031bdd62a0dc30a518c82d9c1a100d6e8420bb2a6f938c48f 1) wiki | commonswiki title | Gnome-edit-delete.svg production_container | wikipedia-commons-local-public.96 production_path | archive/9/96/20120911151633!Gnome-edit-delete.svg sha1 | 36eb92a8d41c46e574f4e54ba352ec53211cdfbc sha256 | e5f3511e773c814662440bd0f43a7f8806323d17c940cd0de8903a153f0927cd size | 46509 type | DRAWING production_status | archived production_url | https://upload.wikimedia.org/wikipedia/commons/archive/9/96/20120911151633%21Gnome-edit-delete.svg upload_date | 2008-01-12 10:43:24 archive_date | 2012-09-11 15:16:33 delete_date | None backup_status | backedup backup_date | 2021-09-19 10:02:47 backup_location | https://backup1007.eqiad.wmnet:9000 backup_container | mediabackups backup_path | commonswiki/e5f/e5f3511e773c814662440bd0f43a7f8806323d17c940cd0de8903a153f0927cd [2024-05-02 13:22:05,595] INFO:query Printed 2 file(s) and finished execution [2024-05-02 13:22:05,595] INFO:query Finishing the interactive query session
This leaves me with very little idea of what happened when. It looks like it ought to be possible to extract the object from backups and upload it to the relevant swift containers, though? I'm not sure if some mw state also needs inspecting / changing first, though.
This icon is used by a bunch of deletion-related templates, , so its absence creates visible breakage. For example, when an admin looks at a page tagged with {{SD}}, it displays a broken-icon instead of the trashcan for the delete function.
Here it is the 2 file versions (with the hash it can be checked they are the same files):
sha1 | 36eb92a8d41c46e574f4e54ba352ec53211cdfbc
sha1 | 3a36632c6569fcdf45aca81a71b53ae4faf80083
Ideally they are uploaded again directly by contributors, as a new version, for recovery rather than we overwritting Swift while loss research continues.
I uploaded both of the recovered file versions under https://commons.wikimedia.org/wiki/File:Gnome-edit-delete.svg to Commons.
This leaves me with very little idea of what happened when. It looks like it ought to be possible to extract the object from backups and upload it to the relevant swift containers, though? I'm not sure if some mw state also needs inspecting / changing first, though.
The full metadata indicates that the backup failed on eqiad only- which is weird (it could be missing by then?). I am checking the logs to see why.
It was failing back in 2021:
[2021-12-11 05:13:49,322] ERROR:swiftclient.service Object GET failed: http://ms-fe.svc.codfw.wmnet/v1/AUTH_mw/wikipedia-commons-local-public.96/9/96/Gnome-edit-delete.svg 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' swiftclient.exceptions.ClientException: Object GET failed: http://ms-fe.svc.codfw.wmnet/v1/AUTH_mw/wikipedia-commons-local-public.96/9/96/Gnome-edit-delete.svg 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' [2021-12-11 05:13:49,323] ERROR:backup '9/96/Gnome-edit-delete.svg' failed to be downloaded [2021-12-11 05:13:49,323] ERROR:backup Download of "commonswiki Gnome-edit-delete.svg 3a36632c6569fcdf45aca81a71b53ae4faf80083" failed
And only very recently was uploaded:
[2024-05-06 14:33:29,993] INFO:backup '9/96/Gnome-edit-delete.svg' downloaded [2024-05-06 14:33:29,994] WARNING:backup Calculated (3a36632c6569fcdf45aca81a71b53ae4faf80083) and queried (36eb92a8d41c46e574f4e54ba352ec53211cdfbc) sha1 checksums are not the same for "Gnome-edit-delete.svg" [2024-05-06 14:33:29,994] INFO:backup sha256 sum of Gnome-edit-delete.svg is a45ec2020e0997a031bdd62a0dc30a518c82d9c1a100d6e8420bb2a6f938c48f [2024-05-06 14:33:30,055] INFO:backup Backup of "commonswiki Gnome-edit-delete.svg 3a36632c6569fcdf45aca81a71b53ae4faf80083 2024-05-06 14:15:28" completed correctly [2024-05-06 14:33:33,903] INFO:backup '9/96/Gnome-edit-delete.svg' downloaded [2024-05-06 14:33:33,904] INFO:backup sha256 sum of Gnome-edit-delete.svg is a45ec2020e0997a031bdd62a0dc30a518c82d9c1a100d6e8420bb2a6f938c48f [2024-05-06 14:33:33,912] WARNING:backup A file with the same sha265 as "commonswiki Gnome-edit-delete.svg 3a36632c6569fcdf45aca81a71b53ae4faf80083 2024-05-06 14:16:14" was already uploaded, skipping.
So it was a "coincidence" that was backed up, and not the other way. I will check the backup logs to see why it was failing.
The burst of DELETE for thumbnails is presumably an editor using the "purge" action on-wiki, in an attempt to fix the problem.
I did not another purge on May 5th, not knowing about this task at the time.
Ignore my last comment, that's the backup of the newly uploaded file. Eqiad backed up on 2021-09-08 21:45:12, I think normally. Codfw has been 404 ing (for both the latest and the archived version) at least since 2021, when the first backup run, but I cannot explain why. Maybe it was primarily used as a thumbnail and that has been kept on Swift until very recently? That would be my best theory.
In any case, other case where long-time used originals disappear.
