An unknown error occurred in storage backend "local-swift-eqiad"
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	Yann
	Jul 3 2023, 7:50 PM

Description

I got an error while uploading files with pywikibot on Commons.
Other files from the same batch uploaded fine: https://commons.wikimedia.org/wiki/File:Quatrevingttreize_03_hugo_128kb.mp3

Error

https://www.archive.org/download/quatrevingttreize_2207_librivox/quatrevingttreize_04_hugo.mp3
WARNING: API error backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
Traceback (most recent call last):
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/librivox.py", line 94, in <module>
    filepage.upload(
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/page/_filepage.py", line 267, in upload
    return self.site.upload(self, source_filename=filename, source_url=url,
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_decorators.py", line 92, in callee
    return fn(self, *args, **kwargs)
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_apisite.py", line 2853, in upload
    return Uploader(self, filepage, **kwargs).upload()
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 133, in upload
    return self._upload(self.ignore_warnings, self.report_success)
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 435, in _upload
    return self.submit(final_request, result, data.get('result'),
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 455, in submit
    result = request.submit()['upload']
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/data/api/_requests.py", line 1088, in submit
    raise pywikibot.exceptions.APIError(**args)
pywikibot.exceptions.APIError: backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
[param: action=upload&filename=Quatrevingttreize+04+hugo.mp3&url=https%3A%2F%2Fwww.archive.org%2Fdownload%2Fquatrevingttreize_2207_librivox%2Fquatrevingttreize_04_hugo.mp3&comment=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&text=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&assert=user&maxlag=5&format=json&token=<redacted>;
 stasherrors: [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', "Path doesn't exist."], 'code': 'uploadstash-exception', 'type': 'error'}];
 servedby: mw1392;
 help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]
CRITICAL: Exiting due to uncaught exception APIError: backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
[param: action=upload&filename=Quatrevingttreize+04+hugo.mp3&url=https%3A%2F%2Fwww.archive.org%2Fdownload%2Fquatrevingttreize_2207_librivox%2Fquatrevingttreize_04_hugo.mp3&comment=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&text=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%271%C3%A9re+partie+-+livre+troisi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&assert=user&maxlag=5&format=json&token=<redacted>;
 stasherrors: [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', "Path doesn't exist."], 'code': 'uploadstash-exception', 'type': 'error'}];
 servedby: mw1392;
 help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]

Impact

File failed to upload. Bot crashed.

Related Objects

Mentioned In: T328872: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw
T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg
Mentioned Here: T328872: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw
T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg
T270866: UploadStashBadPathException 'Path doesn't exist.' when uploading a specific image in Wikimedia Commons via API action=upload

Event Timeline

Yann created this task.Jul 3 2023, 7:50 PM

Restricted Application added a project: Internet-Archive. · View Herald TranscriptJul 3 2023, 7:50 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald Transcript

Yann updated the task description. (Show Details)Jul 3 2023, 7:52 PM

JJMC89 edited projects, added MediaWiki-Action-API; removed Pywikibot, Internet-Archive.Jul 3 2023, 7:57 PM

Again

https://www.archive.org/download/quatrevingttreize_2207_librivox/quatrevingttreize_17_hugo_128kb.mp3
WARNING: API error backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
Traceback (most recent call last):
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/librivox.py", line 94, in <module>
    filepage.upload(
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/page/_filepage.py", line 267, in upload
    return self.site.upload(self, source_filename=filename, source_url=url,
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_decorators.py", line 92, in callee
    return fn(self, *args, **kwargs)
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_apisite.py", line 2853, in upload
    return Uploader(self, filepage, **kwargs).upload()
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 133, in upload
    return self._upload(self.ignore_warnings, self.report_success)
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 435, in _upload
    return self.submit(final_request, result, data.get('result'),
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/site/_upload.py", line 455, in submit
    result = request.submit()['upload']
  File "/cygdrive/c/Users/yannf/OneDrive/Documents/core_stable/pywikibot/data/api/_requests.py", line 1088, in submit
    raise pywikibot.exceptions.APIError(**args)
pywikibot.exceptions.APIError: backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
[param: action=upload&filename=Quatrevingttreize+17+hugo+128kb.mp3&url=https%3A%2F%2Fwww.archive.org%2Fdownload%2Fquatrevingttreize_2207_librivox%2Fquatrevingttreize_17_hugo_128kb.mp3&comment=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&text=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&assert=user&maxlag=5&format=json&token=<redacted>;
 stasherrors: [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', "Path doesn't exist."], 'code': 'uploadstash-exception', 'type': 'error'}];
 servedby: mw1412;
 help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]
CRITICAL: Exiting due to uncaught exception APIError: backend-fail-internal: An unknown error occurred in storage backend "local-swift-eqiad".
[param: action=upload&filename=Quatrevingttreize+17+hugo+128kb.mp3&url=https%3A%2F%2Fwww.archive.org%2Fdownload%2Fquatrevingttreize_2207_librivox%2Fquatrevingttreize_17_hugo_128kb.mp3&comment=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&text=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0A%7B%7BInformation%0A%7CDescription%3D%7B%7Ben%7C%27%27Quatre-vingt-treize%27%27%2C+by+Victor+Hugo%2C+in+French.+LibriVox+recording+by+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7B%7Bfr%7C%27%27Quatre-vingt-treize%27%27%2C+par+Victor+Hugo.+Enregistrement+LibriVox+par+Bidou.+%27%273i%C3%A8me+partie+-+livre+sixi%C3%A8me%27%27.%7D%7D%0A%7CSource%3Dhttps%3A%2F%2Farchive.org%2Fdetails%2Fquatrevingttreize_2207_librivox+https%3A%2F%2Flibrivox.org%2Fquatrevingt-treize-by-victor-hugo%2F%0A%7CDate%3D2022-07-25%0A%7CAuthor%3D%7B%7BCreator%3AVictor+Hugo%7D%7D%0A%7CPermission%3D%0A%7Cother_versions%3D%0A%7D%7D%0A%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0ARecording%3A%0A%7B%7BLibriVox+public+domain%7D%7D%0AOriginal%0A%7B%7BPD-old-100-expired%7D%7D%0A%0A%5B%5BCategory%3ALibriVox+-+Quatre-vingt-treize%2C+by+Victor+Hugo%5D%5D%0A%5B%5BCategory%3AFiles+uploaded+by+Yann+Forget%5D%5D%0A&assert=user&maxlag=5&format=json&token=<redacted>
 stasherrors: [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', "Path doesn't exist."], 'code': 'uploadstash-exception', 'type': 'error'}];
 servedby: mw1412;
 help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]

Umherirrender added a project: SRE-swift-storage.Jul 4 2023, 7:33 PM

It is better to avoid publish your token used on api request (this could be an issue with the debugging in the used framework)

Looks like a duplicate of T270866, while this is a url-upload and T270866 is a file upload, because this is a normal api upload, no stashing, no chunks, no async.

But for url-upload there could be also a timeout in the process on downloading the file, I have read that www.archive.org is often slow on downloads to commons. After the timeout no path exists to process further.

OK, redacted.

Krinkle edited projects, added MediaWiki-Uploading, Unstewarded-production-error; removed MediaWiki-Action-API.Jul 27 2023, 3:15 PM

Krinkle moved this task from Untriaged to Jul 2023 on the Wikimedia-production-error board.

Am I experiencing the same problem?

I'm getting this error using the normal uploader at Special:Upload. After trying to upload, Special:Upload displays a red box saying "An unknown error occurred in storage backend "local-swift-eqiad"." and sometimes the uploaded file is replaced by the original file while the original file disappears.

I've compiled a gallery at https://commons.wikimedia.org/wiki/User:Beao/Images_with_upload_error

Let me know if this should be a new task.

See T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg as well.

Beao mentioned this in T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg.Oct 15 2023, 8:31 PM

I've tried several times to upload File:The New Monthly Magazine - Volume 011.djvu, and had this error most times. For the current version of the file it seemed to reassemble the upload chunks and publish it successfully, but the resulting file seems corrupted: attempting to download it gives a 404 error, it can't render page images, but does seem to be able to extract the text layer (e.g. for pages on Wikisource).

EDIT: Derived images are now working, and the trying to download it no longer gives a 404 error.

This seems to be related to T328872. I get alternatively this error, and the other.

Yann mentioned this in T328872: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw.Oct 19 2023, 4:49 PM

In T341007#9251361, @Beao wrote:

Am I experiencing the same problem?

I'm getting this error using the normal uploader at Special:Upload. After trying to upload, Special:Upload displays a red box saying "An unknown error occurred in storage backend "local-swift-eqiad"." and sometimes the uploaded file is replaced by the original file while the original file disappears.

I've compiled a gallery at https://commons.wikimedia.org/wiki/User:Beao/Images_with_upload_error

Which of these have an original file disappearing? I didn't see 404 errors on the ones that I checked from that list.

It seems like that's been repaired somehow, both thumbnail and file of the original upload was missing for a couple of days for most of them. Now the only issue is the preview not being up to date for a couple of images, I've updated the gallery to reflect that.

That sounds consistent with the operations failing in one data-center (the closest to the uploader) and then getting cleaned up by the periodic sync script.

Sounds reasonable. I was also able to purge the remaining preview images, so now my issue is resolved!

Umherirrender unsubscribed.May 9 2024, 9:23 PM

I got again this error while trying to upload a big PNG file using https://commons.wikimedia.org/wiki/User:Rillke/bigChunkedUpload.js.
Chrome console:

Failed to load resource: the server responded with a status of 403 ()Understand this errorAI
File:Adams_Synchronological_Chart,_1881.png:1 Refused to execute script from 'https://commons.wikimedia.org/w/index.php?title=User:The_Photographer/QICvote.js&action=raw&ctype=text/javascript' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.Understand this errorAI
index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1136 mwbot> stashfailed: Could not store upload in the stash (UploadStashFileException): "An unknown error occurred in storage backend "local-swift-eqiad".". Object success Object

I tried different chunk sizes: 10 MB and 20 MB, and my Upload stash shows:

11:28, 25 February 2025	1blquc7nj7j4.y3epnh.1.png	No thumbnail	0 × 0 (519.09 MB)
12:15, 25 February 2025	1blqxup05ci8.bvryzz.1.png	No thumbnail	0 × 0 (519.09 MB)

I'm not seeing any elevated errors from swift today.

I looked for the first of those two files (sudo cumin -x --force --no-progress --no-color -o txt O:swift::proxy "grep -F '1blqxup05ci8.bvryzz.1.png' /var/log/swift/proxy-access.log") and I find no error codes in the ~125 hits. Broadly, there's a series of PUTs (all of which return 201), then a series of GETs (all of which return 200), then a series of DELETEs (all of which return 204); for objects with names like 1blqxup05ci8.bvryzz.1.png.2 So AFAICT, swift did what it was told, and something higher in the stack uploaded a bunch of fragments, did something with them and then deleted them all.

Why do you think this was a recurrence of the error you reported in 2023?

It worked after I disabled https://commons.wikimedia.org/w/index.php?title=User:The_Photographer/QICvote.js

In T341007#10579276, @MatthewVernon wrote:

I looked for the first of those two files (sudo cumin -x --force --no-progress --no-color -o txt O:swift::proxy "grep -F '1blqxup05ci8.bvryzz.1.png' /var/log/swift/proxy-access.log") and I find no error codes in the ~125 hits. Broadly, there's a series of PUTs (all of which return 201), then a series of GETs (all of which return 200), then a series of DELETEs (all of which return 204); for objects with names like 1blqxup05ci8.bvryzz.1.png.2 So AFAICT, swift did what it was told, and something higher in the stack uploaded a bunch of fragments, did something with them and then deleted them all.

Why do you think this was a recurrence of the error you reported in 2023?

It is the same message, and I got it in the same condition.

In T341007#10579426, @Yann wrote:

In T341007#10579276, @MatthewVernon wrote:

Why do you think this was a recurrence of the error you reported in 2023?

It is the same message, and I got it in the same condition.

It isn't the same message (e.g. this time you have the bit about Refused to execute script from 'https://commons.wikimedia.org/w/index.php?title=User:The_Photographer/QICvote.js&action=raw&ctype=text/javascript' in the backtrace), nor the same condition (in 2023 you were uploading with pywikibot, this time you're using bigchunkedupload.js), and indeed it turned out to be a different solution (disable QICvote.js).

In general, can you open a new ticket, please, rather than commenting on old tickets which look similar? It's much easier to merge tickets than split them.

An unknown error occurred in storage backend "local-swift-eqiad"Closed, ResolvedPublicPRODUCTION ERRORActions