Page MenuHomePhabricator

Export to PDF fails on enwiki
Closed, ResolvedPublic

Description

I was notified on IRC that "Export to PDF" is not functioning on enwiki, and was able to reproduce:

https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=render_article&arttitle=Hey+Monday&returnto=Hey+Monday&oldid=726360553&writer=rdf2latex

cscott (assigned) told me on IRC that he is working on it.

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 19 2016, 4:39 PM
cscott triaged this task as Unbreak Now! priority.Jul 19 2016, 4:43 PM
cscott added a project: Operations.
Restricted Application added subscribers: Luke081515, TerraCodes. · View Herald TranscriptJul 19 2016, 4:43 PM
Gehel added a subscriber: Gehel.Jul 19 2016, 4:43 PM

mark was notified at about 10:58AM EDT. He pinged me on IRC, I started working on it 12:36pm EDT. Ops says no deployments recently, last change to mediawiki-config was "about 14 hours ago". icinga says OCG service is still up and healthy, so I suspect a PHP-side configuration b0rk.

A number of independent confirmations that the OCG servers appear to be healthy and answering queries on port 8000. However, logstash reports "Request to http://ocg.svc.eqiad.wmnet:8000 resulted in error". Looking at ganglia, looks like network traffic to the OCG servers stopped around 1800 UTC on Monday Jul 18. Server Admin Log fingers T140658 as a possible culprit.

Seems that the proxy change caused the urldownloader service not to be able to reach internal hosts, which broken ExtensionDistributor as well (T140753). Doing a revert of T140658 (https://gerrit.wikimedia.org/r/299793) to fix.

It would be nice if we could figure out an icinga tweak that would enable prompter notification of these sorts of failures. None of our health checks appear to use urldownloader.

Confirmed fixed at 1:15:43 PM EDT.

cscott closed this task as Resolved.Jul 19 2016, 5:16 PM