Page MenuHomePhabricator

Special:Upload responds with UploadStashFileNotFoundException "Key .. not found in stash"
Open, Needs TriagePublicPRODUCTION ERROR

Description

Error

Request ID: W6JHNwrAADYAADZMmhgAAABY

message
[{exception_id}] {exception_url} UploadStashFileNotFoundException from line 141 of /srv/mediawiki/php-1.32.0-wmf.22/includes/upload/UploadStash.php: Key "161qx5lra5v4.###.###.jpg" not found in stash.
stacktrace
#0 /srv/mediawiki/php-1.32.0-wmf.22/includes/upload/UploadStash.php(183): UploadStash->getFile(string)
#1 /srv/mediawiki/php-1.32.0-wmf.22/includes/upload/UploadFromStash.php(102): UploadStash->getMetadata(string)
#2 /srv/mediawiki/php-1.32.0-wmf.22/includes/upload/UploadFromStash.php(128): UploadFromStash->initialize(string, string)
#3 /srv/mediawiki/php-1.32.0-wmf.22/includes/upload/UploadBase.php(196): UploadFromStash->initializeFromRequest(WebRequest)
#4 /srv/mediawiki/php-1.32.0-wmf.22/includes/specials/SpecialUpload.php(97): UploadBase::createFromRequest(WebRequest)
#5 /srv/mediawiki/php-1.32.0-wmf.22/includes/specials/SpecialUpload.php(192): SpecialUpload->loadRequest()
#6 /srv/mediawiki/php-1.32.0-wmf.22/includes/specialpage/SpecialPage.php(569): SpecialUpload->execute(NULL)
#7 /srv/mediawiki/php-1.32.0-wmf.22/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run(NULL)
#8 /srv/mediawiki/php-1.32.0-wmf.22/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#9 /srv/mediawiki/php-1.32.0-wmf.22/includes/MediaWiki.php(868): MediaWiki->performRequest()
#10 /srv/mediawiki/php-1.32.0-wmf.22/includes/MediaWiki.php(525): MediaWiki->main()
#11 /srv/mediawiki/php-1.32.0-wmf.22/index.php(42): MediaWiki->run()

Notes

Recorded 149 times in WMF Logstash over the last 30 days.

  • Domains: commons.wikimedia.org, mediawiki.org, en.wikipedia.org and my.wikipedia.org
  • Versions: 1.32.0-wmf.16, 1.32.0-wmf.18, 1.32.0-wmf.19, 1.32.0-wmf.20, 1.32.0-wmf.22.

This error has been happening for a while, filing retroactively.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM
daniel subscribed.

Seen again today:

#0 /srv/mediawiki/php-1.36.0-wmf.6/includes/upload/UploadStash.php(177): UploadStash->getFile(string)
#1 /srv/mediawiki/php-1.36.0-wmf.6/includes/upload/UploadFromStash.php(101): UploadStash->getMetadata(string)
#2 /srv/mediawiki/php-1.36.0-wmf.6/includes/upload/UploadFromStash.php(127): UploadFromStash->initialize(string, string)
#3 /srv/mediawiki/php-1.36.0-wmf.6/includes/upload/UploadBase.php(221): UploadFromStash->initializeFromRequest(WebRequest)
#4 /srv/mediawiki/php-1.36.0-wmf.6/includes/specials/SpecialUpload.php(97): UploadBase::createFromRequest(WebRequest)
#5 /srv/mediawiki/php-1.36.0-wmf.6/includes/specials/SpecialUpload.php(201): SpecialUpload->loadRequest()
#6 /srv/mediawiki/php-1.36.0-wmf.6/includes/specialpage/SpecialPage.php(600): SpecialUpload->execute(NULL)
#7 /srv/mediawiki/php-1.36.0-wmf.6/includes/specialpage/SpecialPageFactory.php(680): SpecialPage->run(NULL)
#8 /srv/mediawiki/php-1.36.0-wmf.6/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#9 /srv/mediawiki/php-1.36.0-wmf.6/includes/MediaWiki.php(940): MediaWiki->performRequest()
#10 /srv/mediawiki/php-1.36.0-wmf.6/includes/MediaWiki.php(543): MediaWiki->main()
#11 /srv/mediawiki/php-1.36.0-wmf.6/index.php(53): MediaWiki->run()
#12 /srv/mediawiki/php-1.36.0-wmf.6/index.php(46): wfIndexMain()
#13 /srv/mediawiki/w/index.php(3): require(string)
#14 {main}

Request: 5e49565a-ffb4-47aa-bb2f-71dc25044927
Wiki: ar.wikipedia.org

IMO it's quite likely that these are simply (very late) resubmissions from Special:Upload after the original upload failed (in which case the file got stashed), but the resubmission took long enough that the stashed file got cleaned up in the meantime.
See T38587#6267136 (duplicate) for more details.

IDK what (if anything) we want to do about it in that case:

  • wait longer before clearing stash? (is the additional storage worth it?)
  • stop treating this as an exception? (although we may not be able to differentiate between "file doesn't exist because of an actual issue" and "file got cleaned up")
  • alter Special:Upload to prevent POSTing a page older than <time between cleanups>? (that value would live in different repos and might get out of sync)
  • ignore this?

IDK what (if anything) we want to do about it in that case:

  • wait longer before clearing stash? (is the additional storage worth it?)

That would not really solve the issue, just make it less common, right?
It seems to me that expected behavior should never cause an error to be logged.

  • stop treating this as an exception? (although we may not be able to differentiate between "file doesn't exist because of an actual issue" and "file got cleaned up")

We may want to know at least when this issue spikes... doe we have tooling in place that would allow us to ignore some warning unless it exceeds a certain volume?

  • alter Special:Upload to prevent POSTing a page older than <time between cleanups>? (that value would live in different repos and might get out of sync)

This sounds like the best option. Ensuring consistency of config between services is a problem that should be tackled separately.