MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	• thcipriani
	Jul 2 2026, 3:49 PM

Project Tags

Referenced Files

	F96122570: image.png
	Mon, Jul 27, 1:37 PM

Subscribers

Description

Error

service.version: 1.47.0-wmf.9
timestamp: 2026-07-02T13:38:43.499Z
labels.phpversion: 8.3.31
trace.id: a4b3cc9d-ae54-4383-a150-7ff8d29fc15e
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw

Frame	Location	Call
from	/srv/mediawiki/php-1.47.0-wmf.9/includes/Upload/UploadFromChunks.php(441)
#0	/srv/mediawiki/php-1.47.0-wmf.9/includes/Upload/UploadFromChunks.php(342)	MediaWiki\Upload\UploadFromChunks->outputChunk(string)
#1	/srv/mediawiki/php-1.47.0-wmf.9/includes/Api/ApiUpload.php(428)	MediaWiki\Upload\UploadFromChunks->addChunk(string, int, int)
#2	/srv/mediawiki/php-1.47.0-wmf.9/includes/Api/ApiUpload.php(296)	MediaWiki\Api\ApiUpload->getChunkResult(array)
#3	/srv/mediawiki/php-1.47.0-wmf.9/includes/Api/ApiUpload.php(171)	MediaWiki\Api\ApiUpload->getContextResult()
#4	/srv/mediawiki/php-1.47.0-wmf.9/includes/Api/ApiMain.php(2049)	MediaWiki\Api\ApiUpload->execute()
#5	/srv/mediawiki/php-1.47.0-wmf.9/includes/Api/ApiMain.php(947)	MediaWiki\Api\ApiMain->executeAction()
#6	/srv/mediawiki/php-1.47.0-wmf.9/includes/Api/ApiMain.php(918)	MediaWiki\Api\ApiMain->executeActionWithErrorHandling()
#7	/srv/mediawiki/php-1.47.0-wmf.9/includes/Api/ApiEntryPoint.php(138)	MediaWiki\Api\ApiMain->execute()
#8	/srv/mediawiki/php-1.47.0-wmf.9/includes/MediaWikiEntryPoint.php(180)	MediaWiki\Api\ApiEntryPoint->execute()
#9	/srv/mediawiki/php-1.47.0-wmf.9/api.php(30)	MediaWiki\MediaWikiEntryPoint->run()
#10	/srv/mediawiki/w/api.php(3)	require(string)
#11	{main}

Notes

Happens 10×/day in production on commons upload
Unsure the right team to flag this for, flagged the infra and core teams, retag as appropriate!

Details

Request URL: https://commons.wikimedia.org/w/api.php?action=upload

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Upload: Do not throw for failure to save a chunk file	Ladsgroup	mediawiki/core	wmf/1.47.0-wmf.11	+23 -1
	Upload: Do not throw for failure to save a chunk file	Samwilson	mediawiki/core	master	+23 -1

Customize query in gerrit

Related Objects

Mentioned In: T428797: [wmf.5-regression] Add image dialog doesn't display image
Mentioned Here: T428797: [wmf.5-regression] Add image dialog doesn't display image

Event Timeline

• thcipriani created this task.Jul 2 2026, 3:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 2 2026, 3:49 PM

• thcipriani moved this task from Untriaged to Jul–Sep 2026 on the Wikimedia-production-error board.Jul 2 2026, 3:49 PM

The more useful message is logged in the FileOperation channel, which has its own dashboard. (The exception should probably contain it somehow.) It's mostly Swift HTTP 503s, with a few 504s thrown in. So I don't think much can be done here on the MediaWiki side.

Umherirrender added a project: SRE-swift-storage.Jul 3 2026, 2:57 PM

JTweed-WMF edited projects, added: MediaWiki-Media-Platform-Team; removed: MediaWiki-Core-Platform-Team.Jul 6 2026, 2:37 PM

LSobanski removed a project: Infrastructure-Foundations.Jul 7 2026, 12:12 PM

JTweed-WMF triaged this task as Medium priority.Jul 9 2026, 9:43 AM

JTweed-WMF lowered the priority of this task from Medium to Low.

JTweed-WMF moved this task from Inbox (Untriaged) to Backlog (Triaged) on the MediaWiki-Media-Platform-Team board.

Umherirrender merged a task: T431809: Caught exception of type MediaWiki\Upload\Exception\UploadChunkFileException.Fri, Jul 10, 5:48 PM

Umherirrender added a subscriber: Yann.

There's probably nothing to be done to fix the actual failure on the Swift side, but UploadApi returns stashfailed for other issues, so perhaps should for this error also.

Change #1311310 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/core@master] Upload: Do not throw for chunk output failures

https://gerrit.wikimedia.org/r/1311310

gerritbot added a project: Patch-For-Review.Thu, Jul 16, 6:39 AM

Samwilson moved this task from In Progress to Code Review on the MediaWiki-Media-Platform-Team board.Thu, Jul 16, 7:57 AM

Change #1311420 had a related patch set uploaded (by Ladsgroup; author: Samwilson):

[mediawiki/core@wmf/1.47.0-wmf.11] Upload: Do not throw for failure to save a chunk file

https://gerrit.wikimedia.org/r/1311420

Change #1311310 merged by jenkins-bot:

[mediawiki/core@master] Upload: Do not throw for failure to save a chunk file

https://gerrit.wikimedia.org/r/1311310

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.12; 2026-07-21).Thu, Jul 16, 8:00 PM

Change #1311420 merged by jenkins-bot:

[mediawiki/core@wmf/1.47.0-wmf.11] Upload: Do not throw for failure to save a chunk file

https://gerrit.wikimedia.org/r/1311420

Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:17:32Z] <ladsgroup@deploy2003> Started scap sync-world: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]]

Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:19:19Z] <ladsgroup@deploy2003> ladsgroup, urbanecm: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:26:23Z] <ladsgroup@deploy2003> Finished scap sync-world: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] (duration: 08m 51s)

Maintenance_bot removed a project: Patch-For-Review.Thu, Jul 16, 10:31 PM

ReleaseTaggerBot edited projects, added: MW-1.47-notes (1.47.0-wmf.11; 2026-07-14); removed: MW-1.47-notes (1.47.0-wmf.12; 2026-07-21).Thu, Jul 16, 11:00 PM

Samwilson moved this task from Code Review to To Verify on the MediaWiki-Media-Platform-Team board.Mon, Jul 20, 9:15 AM

This is not throwing any more. (The failures are still being logged, but the front end can retry in the normal way now.)

Yann unsubscribed.Mon, Jul 27, 5:08 PM

MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfwClosed, ResolvedPublicPRODUCTION ERRORActions