An unknown error occurred in storage backend "local-swift-eqiad"
Open, Needs TriagePublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Yann
	Jul 3 2023, 7:50 PM

Description

I got an error while uploading files with pywikibot on Commons.
Other files from the same batch uploaded fine: https://commons.wikimedia.org/wiki/File:Quatrevingttreize_03_hugo_128kb.mp3

Error

https://www.archive.org/download/quatrevingttreize_2207_librivox/quatrevingttreize_04_hugo.mp3
WARNING: API error backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
Traceback (most recent call last):
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/librivox.py", line 94, in <module>
    filepage.upload(
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/page/_filepage.py", line 267, in upload
    return self.site.upload(self, source_filename=filename, source_url=url,
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_decorators.py", line 92, in callee
    return fn(self, *args, **kwargs)
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_apisite.py", line 2853, in upload
    return Uploader(self, filepage, **kwargs).upload()
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 133, in upload
    return self._upload(self.ignore_warnings, self.report_success)
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 435, in _upload
    return self.submit(final_request, result, data.get('result'),
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 455, in submit
    result = request.submit()['upload']
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/data/api/_requests.py", line 1088, in submit
    raise pywikibot.exceptions.APIError(**args)
pywikibot.exceptions.APIError: backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
[param: action=upload&filename=Quatrevingttreize+04+hugo.mp3&url=https%3A%2F%2Fwww.archive.org%2Fdownload%2Fquatrevingttreize_2207_librivox%2Fquatrevingttreize_04_hugo.mp3&comment=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&text=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&assert=user&maxlag=5&format=json&token=<redacted>;
 stasherrors: [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', "Path doesn't exist."], 'code': 'uploadstash-exception', 'type': 'error'}];
 servedby: mw1392;
 help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]
CRITICAL: Exiting due to uncaught exception APIError: backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
[param: action=upload&filename=Quatrevingttreize+04+hugo.mp3&url=https%3A%2F%2Fwww.archive.org%2Fdownload%2Fquatrevingttreize_2207_librivox%2Fquatrevingttreize_04_hugo.mp3&comment=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&text=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&assert=user&maxlag=5&format=json&token=<redacted>;
 stasherrors: [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', "Path doesn't exist."], 'code': 'uploadstash-exception', 'type': 'error'}];
 servedby: mw1392;
 help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]

Impact

File failed to upload. Bot crashed.

Related Objects

Mentioned In: T328872: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw
T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg
Mentioned Here: T328872: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw
T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg
T270866: UploadStashBadPathException 'Path doesn't exist.' when uploading a specific image in Wikimedia Commons via API action=upload

Event Timeline

Yann created this task.Jul 3 2023, 7:50 PM

Restricted Application added a project: Internet-Archive. · View Herald TranscriptJul 3 2023, 7:50 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald Transcript

Yann updated the task description. (Show Details)Jul 3 2023, 7:52 PM

JJMC89 edited projects, added MediaWiki-Action-API; removed Pywikibot, Internet-Archive.Jul 3 2023, 7:57 PM

Again

https://www.archive.org/download/quatrevingttreize_2207_librivox/quatrevingttreize_17_hugo_128kb.mp3
WARNING: API error backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
Traceback (most recent call last):
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/librivox.py", line 94, in <module>
    filepage.upload(
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/page/_filepage.py", line 267, in upload
    return self.site.upload(self, source_filename=filename, source_url=url,
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_decorators.py", line 92, in callee
    return fn(self, *args, **kwargs)
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_apisite.py", line 2853, in upload
    return Uploader(self, filepage, **kwargs).upload()
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 133, in upload
    return self._upload(self.ignore_warnings, self.report_success)
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 435, in _upload
    return self.submit(final_request, result, data.get('result'),
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 455, in submit
    result = request.submit()['upload']
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/data/api/_requests.py", line 1088, in submit
    raise pywikibot.exceptions.APIError(**args)
pywikibot.exceptions.APIError: backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
[param: action=upload&filename=Quatrevingttreize+17+hugo+128kb.mp3&url=https%3A%2F%2Fwww.archive.org%2Fdownload%2Fquatrevingttreize_2207_librivox%2Fquatrevingttreize_17_hugo_128kb.mp3&comment=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&text=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&assert=user&maxlag=5&format=json&token=<redacted>;
 stasherrors: [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', "Path doesn't exist."], 'code': 'uploadstash-exception', 'type': 'error'}];
 servedby: mw1412;
 help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]
CRITICAL: Exiting due to uncaught exception APIError: backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
[param: action=upload&filename=Quatrevingttreize+17+hugo+128kb.mp3&url=https%3A%2F%2Fwww.archive.org%2Fdownload%2Fquatrevingttreize_2207_librivox%2Fquatrevingttreize_17_hugo_128kb.mp3&comment=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&text=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&assert=user&maxlag=5&format=json&token=<redacted>
 stasherrors: [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', "Path doesn't exist."], 'code': 'uploadstash-exception', 'type': 'error'}];
 servedby: mw1412;
 help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]

Umherirrender added a project: SRE-swift-storage.Jul 4 2023, 7:33 PM

It is better to avoid publish your token used on api request (this could be an issue with the debugging in the used framework)

Looks like a duplicate of T270866, while this is a url-upload and T270866 is a file upload, because this is a normal api upload, no stashing, no chunks, no async.

But for url-upload there could be also a timeout in the process on downloading the file, I have read that www.archive.org is often slow on downloads to commons. After the timeout no path exists to process further.

OK, redacted.

Krinkle edited projects, added MediaWiki-Uploading, Unstewarded-production-error; removed MediaWiki-Action-API.Jul 27 2023, 3:15 PM

Krinkle moved this task from Untriaged to Jul 2023 on the Wikimedia-production-error board.

Am I experiencing the same problem?

I'm getting this error using the normal uploader at Special:Upload. After trying to upload, Special:Upload displays a red box saying "An unknown error occurred in storage backend "local-swift-eqiad"." and sometimes the uploaded file is replaced by the original file while the original file disappears.

I've compiled a gallery at https://commons.wikimedia.org/wiki/User:Beao/Images_with_upload_error

Let me know if this should be a new task.

See T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg as well.

Beao mentioned this in T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg.Oct 15 2023, 8:31 PM

I've tried several times to upload File:The New Monthly Magazine - Volume 011.djvu, and had this error most times. For the current version of the file it seemed to reassemble the upload chunks and publish it successfully, but the resulting file seems corrupted: attempting to download it gives a 404 error, it can't render page images, but does seem to be able to extract the text layer (e.g. for pages on Wikisource).

EDIT: Derived images are now working, and the trying to download it no longer gives a 404 error.

This seems to be related to T328872. I get alternatively this error, and the other.

Yann mentioned this in T328872: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw.Oct 19 2023, 4:49 PM

In T341007#9251361, @Beao wrote:

Am I experiencing the same problem?

I'm getting this error using the normal uploader at Special:Upload. After trying to upload, Special:Upload displays a red box saying "An unknown error occurred in storage backend "local-swift-eqiad"." and sometimes the uploaded file is replaced by the original file while the original file disappears.

I've compiled a gallery at https://commons.wikimedia.org/wiki/User:Beao/Images_with_upload_error

Which of these have an original file disappearing? I didn't see 404 errors on the ones that I checked from that list.

It seems like that's been repaired somehow, both thumbnail and file of the original upload was missing for a couple of days for most of them. Now the only issue is the preview not being up to date for a couple of images, I've updated the gallery to reflect that.

That sounds consistent with the operations failing in one data-center (the closest to the uploader) and then getting cleaned up by the periodic sync script.

Sounds reasonable. I was also able to purge the remaining preview images, so now my issue is resolved!

Umherirrender unsubscribed.Thu, May 9, 9:23 PM

An unknown error occurred in storage backend "local-swift-eqiad"Open, Needs TriagePublicPRODUCTION ERRORActions