mark was notified at about 10:58AM EDT. He pinged me on IRC, I started working on it 12:36pm EDT. Ops says no deployments recently, last change to mediawiki-config was "about 14 hours ago". icinga says OCG service is still up and healthy, so I suspect a PHP-side configuration b0rk.

A number of independent confirmations that the OCG servers appear to be healthy and answering queries on port 8000. However, logstash reports "Request to http://ocg.svc.eqiad.wmnet:8000 resulted in error". Looking at ganglia, looks like network traffic to the OCG servers stopped around 1800 UTC on Monday Jul 18. Server Admin Log fingers T140658 as a possible culprit.

Seems that the proxy change caused the urldownloader service not to be able to reach internal hosts, which broken ExtensionDistributor as well (T140753). Doing a revert of T140658 (https://gerrit.wikimedia.org/r/299793) to fix.

It would be nice if we could figure out an icinga tweak that would enable prompter notification of these sorts of failures. None of our health checks appear to use urldownloader.

Confirmed fixed at 1:15:43 PM EDT.