UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw
Open, Needs TriagePublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	brennen
	May 22 2025, 3:33 PM

Description

Error

service.version: 1.45.0-wmf.2
trace.id: 718f9058-c5ad-4f6e-a5d9-0022f4754dd6
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw

Frame	Location	Call
from	/srv/mediawiki/php-1.45.0-wmf.2/includes/upload/UploadFromChunks.php(449)
#0	/srv/mediawiki/php-1.45.0-wmf.2/includes/upload/UploadFromChunks.php(350)	UploadFromChunks->outputChunk(string)
#1	/srv/mediawiki/php-1.45.0-wmf.2/includes/api/ApiUpload.php(388)	UploadFromChunks->addChunk(string, int, int)
#2	/srv/mediawiki/php-1.45.0-wmf.2/includes/api/ApiUpload.php(256)	MediaWiki\Api\ApiUpload->getChunkResult(array)
#3	/srv/mediawiki/php-1.45.0-wmf.2/includes/api/ApiUpload.php(173)	MediaWiki\Api\ApiUpload->getContextResult()
#4	/srv/mediawiki/php-1.45.0-wmf.2/includes/api/ApiMain.php(2010)	MediaWiki\Api\ApiUpload->execute()
#5	/srv/mediawiki/php-1.45.0-wmf.2/includes/api/ApiMain.php(948)	MediaWiki\Api\ApiMain->executeAction()
#6	/srv/mediawiki/php-1.45.0-wmf.2/includes/api/ApiMain.php(919)	MediaWiki\Api\ApiMain->executeActionWithErrorHandling()
#7	/srv/mediawiki/php-1.45.0-wmf.2/includes/api/ApiEntryPoint.php(152)	MediaWiki\Api\ApiMain->execute()
#8	/srv/mediawiki/php-1.45.0-wmf.2/includes/MediaWikiEntryPoint.php(198)	MediaWiki\Api\ApiEntryPoint->execute()
#9	/srv/mediawiki/php-1.45.0-wmf.2/api.php(44)	MediaWiki\MediaWikiEntryPoint->run()
#10	/srv/mediawiki/w/api.php(3)	require(string)
#11	{main}

Notes

Noted a few of these during log triage. Not happening at a high volume.

Details

Request URL: https://commons.wikimedia.org/w/api.php

Event Timeline

brennen created this task.May 22 2025, 3:33 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 22 2025, 3:33 PM

I had a look in the swift logs for the associated item (per logstash, https://ms-fe.svc.codfw.wmnet/v1/AUTH_mw/wikipedia-commons-local-temp.85/8/85/1bszz9c7w23k.h5b3mo.6438344.webm.5), and get exactly one hit, in eqiad:

May 21 16:51:30 ms-fe1013 proxy-server: 10.67.188.58 10.64.48.149 21/May/2025/16/51/30 PUT /v1/AUTH_mw/wikipedia-commons-local-temp.85/8/85/1bszz9c7w23k.h5b3mo.6438344.webm.5 HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk3261723f1... 16777216 - d5807b8ad0ae1b0af9f9cf957a42740c txd4becbe641444c8ba1f63-00682e0492 - 0.1807 - - 1747846290.054478645 1747846290.235161781 0

...so that request took 0.1807 seconds in eqiad, and was of 16,777,216 bytes.

There's no equivalent log entry in codfw, and no obvious error in server.log at 16:51:30. The envoy logs for that time don't show any obvious misbehaviour either.

brennen moved this task from Backlog to Logs/Train on the User-brennen board.May 28 2025, 11:42 PM

thcipriani moved this task from Untriaged to Apr-Jun 2025 on the Wikimedia-production-error board.Jul 29 2025, 6:48 PM

There are three more such entries today in Logstash, all around Aug 26, 2025 @ 10:02:40.

UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfwOpen, Needs TriagePublicPRODUCTION ERRORActions