Only Wikipedia Wiki are affected, It works in other projects https://en.wiktionary.org/wiki/File:United_States_change_frame_1870-07-15.png

It's a 1.26wmf5 vs 1.26wmf6 bug

Adding @greg so that he's aware of this issue.

This issue is pretty bad, as it affects transclusions too, which means affected images are actually unusable. Example: https://en.wikipedia.org/wiki/User:Deskana/sandbox

In T99761#1298302, @Reedy wrote:

It's a 1.26wmf5 vs 1.26wmf6 bug

There are three core commits in that range which are obviously file-handling-related:
https://gerrit.wikimedia.org/r/#/c/206968
https://gerrit.wikimedia.org/r/#/c/207004
https://gerrit.wikimedia.org/r/#/c/210091
which are all about multidatacenter-aware caching and seem harmless (even if caching got broken somehow, file lookups would just fall back to querying the DB).

I tried to manually retrace the steps taken when calling wfFindFile:

tgr@terbium:~$ mwscript eval.php --wiki=enwiki
> $t = File::normalizeTitle( 'File:United_States_change_frame_1870-07-15.png' );
> $commons = RepoGroup::singleton()->getRepoByName( 'shared' );
> $f = $commons->newFile( $t );
> $dbr = $commons->getMasterDB();
> echo $dbr->getWikiId();

// $row = $dbr->selectRow( 'image', $this->getCacheFields( 'img_' ), array( 'img_name' => $this->getName() ), $fname );
> echo $dbr->getWikiId();
commonswiki
> echo $dbr->selectSQLText( 'image', $f->getCacheFields( 'img_' ), array( 'img_name' => $f->getName() ), 'eval.php' );
SELECT  img_size,img_width,img_height,img_bits,img_media_type,img_major_mime,img_minor_mime,img_metadata,img_timestamp,img_sha1,img_user,img_user_text,img_description  FROM `image`   WHERE img_name = 'File:United_States_change_frame_1870-07-15.png'

Note the file prefix in the img_name condition; that should not be there.

This is not specific to the wikipedias so not sure if it's causing the current problem or it's a sleeping bug in an unused code path.

This is a cache issue; RepoGroup::singleton()->getRepoByName( 'shared' )->findFile( <title> ) fails but RepoGroup::singleton()->getRepoByName( 'shared' )->findFile( <title>, array( 'latest' => true ) ) works (and fixes that specific file for all wikis).

I can reproduce the error by looking at the would-be description page on enwiki first before uploading on commons. File lookups are cached for 24h even if the file does not exist; with the multiple datacenter support cache semantics have changed and set() does not reliable overwrite an already cached value anymore. I'm guessing cache records for non-existing files need a delete() call when the file is uploaded.

I think there should be at least a Commons site notice to warn of this crucial bug ("Priority: normal" must be a joke, right?). It's a really bad user experience to upload stuff and then not be able to use it on the home wikipedia then, without knowing why at all. It took me half an hour to find this bug report...

LocalFile::recordUpload2() does call delete(). I also tried to call it manually and it made no difference. Also, caching seems to work fine (with a few minutes of delay) but the file still does not show up:

tgr@terbium:~$ mwscript eval.php --wiki=enwiki
> $f = RepoGroup::singleton()->getRepoByName( 'shared' )->newFile( 'Tgr_test_12345.png' );
> $key = $f->getCacheKey();
> $cache = ObjectCache::getMainWANInstance();
> $v = $cache->get( $key );
> echo $v['fileExists'];
1

@AndreasPraefcke, it only affects a fraction of new uploads as far as I could see.

@Tgr: Ah, thanks for the info. I affected all three of my uploads that I tried to use on de.wikipedia today, hence I thought it was really very urgent.

Please stop playing with the priority up and down and let @greg or @Aklapper deal with internal WMF priorities. We all agree it is important and a high severity, the exact priority level is less important.

After today's deploy this is expected to not be a problem.

@aaron for posterity's sake, it would be nice to know WHY after today's deploy it's no longer a problem :)

In T99761#1300035, @TheDJ wrote:

@aaron for posterity's sake, it would be nice to know WHY after today's deploy it's no longer a problem :)

One version used the WAN cache while the other used $wgMemc.