So they are stored in operations-mediawiki-config and the logos are present there https://github.com/wikimedia/operations-mediawiki-config/blob/master/w/static/images/project-logos/wikidatawiki.png

I guess they fell off the server? O_o I guess some ops person needs to poke this.

https://meta.wikimedia.org/w/index.php?title=Www.wikimedia.org_template/temp&diff=12487372&oldid=12374377 if copied to the real template should work around it for wikidata. I'm on my phone but a deployer could try logging in to a random mw host and checking for the missing images, sync-dir w/static/images/project-logos on tin otherwise?

Seems to have resolved itself..?

Or not. It seems to either be related to what machine I try to download the file from (e.g. I get an image on tin and terbium, but hooft gets a 404). My laptop gets an image, but I was getting a 404 at college earlier.

Ok, it seems related to the www.wikimedia.org apache config, in fact directly on an appserver I get:

$ curl -H 'Host: en.wikipedia.org' localhost/static/images/project-logos/enwikiversity.png -I
HTTP/1.1 200 OK
Date: Mon, 22 Jun 2015 15:37:31 GMT
Server: Apache
X-Powered-By: HHVM/3.3.0-static
Last-Modified: Thu, 07 May 2015 20:03:54 GMT
ETag: "24e6-51583693c1e80"
Accept-Ranges: bytes
Content-Length: 9446
Cache-Control: max-age=31536000
Expires: Tue, 21 Jun 2016 15:37:31 GMT
Access-Control-Allow-Origin: *
Content-Type: image/png

oblivian@mw1018:~$ curl -H 'Host: www.wikipedia.org' localhost/static/images/project-logos/enwikiversity.png -I
HTTP/1.1 301 Moved Permanently
Date: Mon, 22 Jun 2015 15:37:39 GMT
Server: Apache
X-Powered-By: HHVM/3.3.0-static
Location: http://en.wikipedia.org/static/images/project-logos/enwikiversity.png
Content-Type: text/html; charset=iso-8859-1

oblivian@mw1018:~$ curl -H 'Host: www.wikimedia.org' localhost/static/images/project-logos/enwikiversity.png -I
HTTP/1.1 404 Not Found
Date: Mon, 22 Jun 2015 15:37:52 GMT
Server: Apache
X-Powered-By: HHVM/3.6.1
Cache-Control: s-maxage=2678400, max-age=2678400
X-Wikimedia-Debug: prot=http:// serv=www.wikimedia.org loc=/static/images/project-logos/enwikiversity.png
Refresh: 5; url=http://www.wikimedia.org/wiki/static/images/project-logos/enwikiversity.png
Content-Type: text/html; charset=utf-8

The X-Powered-By line suggests that while enwiki vhost correctly uses the static file and www.wikipedia.org does a simple redirect, www.wikimedia.org does not.

I suspect this has to do with redirects.dat.

we still see the other logos because they've been cached, apparently.

I'm not sure what changed here, to be honest.

Found it:

$ ls /srv/mediawiki/docroot/wwwportal

w/

there is no link to the static directory there, it must have been removed in some deploy, maybe eons ago. We noticed now as the cache on those objects has expired in the meanwhile

It doesn't look like the a static link directory existed in mediawiki-config at docroot/wwwportal before, but it does seem like the template has been edited somewhat recently:

https://meta.wikimedia.org/w/index.php?title=Www.wikimedia.org_template&diff=12374381&oldid=12369391

so it looks like if you revert to this revision:
https://meta.wikimedia.org/w/index.php?title=Www.wikimedia.org_template&oldid=12369391

it should fix the problem.

I, evidently, don't have permission to edit this page:

You do not have permission to edit this page, for the following reason:

This page has been protected to prevent editing or other actions.

In T103296#1389234, @thcipriani wrote:

so it looks like if you revert to this revision:
https://meta.wikimedia.org/w/index.php?title=Www.wikimedia.org_template&oldid=12369391

it should fix the problem.

I, evidently, don't have permission to edit this page:

You do not have permission to edit this page, for the following reason:

This page has been protected to prevent editing or other actions.

It would work around the issue, yes, and I could, but please let's not.

Well...
https://meta.wikimedia.org/w/index.php?title=Www.wikimedia.org_template&diff=next&oldid=12369391
The change broke it, thus it should be reverted no?

https://www.wikimedia.org/static/images/project-logos/enwikiversity.png Failed to load resource: the server responded with a status of 404 (Not Found)
https://www.wikimedia.org/static/images/project-logos/wikidatawiki.png Failed to load resource: the server responded with a status of 404 (Not Found)

These now work. Probably because someone tried to access them meanwhile via a different hostname.

Requests for /static/ are cached in Varnish for text domains irregardless of the hostname because it is the same directory in all docroots. In case of cache miss, it will populate it from the specified hostname, however.

Most likely the other icons were all requested before, but not these two. And because they then fallback to the actual domain, the lack of static/ on www.wikimedia.org (and other www portals) was exposed.

Fixed now with 8c7e7131a0e823.