@Tgr: I tried to reproduce, but it seems to work now. Can you confirm?

Urls I tried:
http://upload.wikimedia.org/wikipedia/commons/thumb/5/52/Giant_planes_comparison.svg/11px-Giant_planes_comparison.svg.png
and
http://upload.wikimedia.org/wikipedia/commons/thumb/5/52/Giant_planes_comparison.svg/121px-Giant_planes_comparison.svg.png

(and associated api calls before)

The bug only affects beta upload URLs (upload.beta.wmflabs.org) not production ones (upload.wikimedia.org). I can still reproduce it with such URLs. There was a copy-paste error in the description, now fixed; sorry about that.

I have no idea what decides whether the api returns a beta or a production URL; it seems to be random. I think the beta one is correct since beta enwiki uses $wgUseInstantCommons and that has thumbnail caching enabled.

I think this is not beta-related but a bug (or quirk) of how thumb.php works.

What happens is basically that thumb.php does not do remote file fetches (T27958); it returns a 404 instead, and that 404 gets cached in varnish. Afterwards the API is queried for the same file, fetches it and generates the thumbnail, but at this point the 404 is already cached in varnish.

This does not affect production because it has not remote repos configured (Commons is a shared-DB repo, and that behaves as a local repo as far as file handling is concerned).

There is still the issue of the API sometimes returning a local and sometimes a remote thumbnail; compare

http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&prop=imageinfo&format=jsonfm&titles=File:Giant_planes_comparison.svg&iiprop=url&iiurlwidth=129
http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&prop=imageinfo&format=jsonfm&titles=File:Giant_planes_comparison.svg&iiprop=url&iiurlwidth=130

Maybe something got stuck in memcached or varnish?

I'm wondering about moving this from Beta-Cluster-Infrastructure to Beta-Cluster-reproducible... But I did spot this in /data/project/upload7/scripts/thumb-handler.php (the script nginx on deployment-upload runs to handle upload.beta.wmflabs.org requests):

} elseif ( $httpCode == 404 ) {
        header( 'HTTP/1.1 404 Not found' );
        // header( 'Cache-Control: no-cache' );
        header( 'Cache-Control: s-maxage=300, must-revalidate, max-age=0' );