Around 13:50 UTC, a whole bunch of webservices decided to stop. Until 13:49:14 it's all wikiviewstats (T122506), but then we get:
```
0 2015-12-28 13:49:28 tools-webgrid-lighttpd-1206.eqiad.wmflabs tools.rxy lighttpd-rxy 261226
0 2015-12-28 13:49:32 tools-webgrid-lighttpd-1410.eqiad.wmflabs tools.expose-data lighttpd-expose-data 257452
0 2015-12-28 13:49:35 tools-webgrid-lighttpd-1407.eqiad.wmflabs tools.bibleversefinder lighttpd-bibleversefinder 257453
0 2015-12-28 13:49:38 tools-webgrid-lighttpd-1407.eqiad.wmflabs tools.raun lighttpd-raun 257455
0 2015-12-28 13:49:42 tools-webgrid-lighttpd-1406.eqiad.wmflabs tools.contributionsurveyor lighttpd-contributionsurveyor 257456
0 2015-12-28 13:49:46 tools-webgrid-lighttpd-1208.eqiad.wmflabs tools.jembot lighttpd-jembot 403097
0 2015-12-28 13:49:50 tools-webgrid-lighttpd-1205.eqiad.wmflabs tools.usergraph lighttpd-usergraph 261227
0 2015-12-28 13:49:54 tools-webgrid-lighttpd-1408.eqiad.wmflabs tools.whois lighttpd-whois 257467
0 2015-12-28 13:49:56 tools-webgrid-lighttpd-1201.eqiad.wmflabs tools.wle lighttpd-wle 257468
0 2015-12-28 13:49:59 tools-webgrid-lighttpd-1409.eqiad.wmflabs tools.twl lighttpd-twl 257469
0 2015-12-28 13:50:02 tools-webgrid-lighttpd-1205.eqiad.wmflabs tools.avicbot twitter.sh 484979
0 2015-12-28 13:50:03 tools-webgrid-lighttpd-1414.tools.eqiad.wmflabs tools.voiceintro lighttpd-voiceintro 257470
0 2015-12-28 13:50:07 tools-webgrid-lighttpd-1207.eqiad.wmflabs tools.supercount lighttpd-supercount 257471
0 2015-12-28 13:50:10 tools-webgrid-lighttpd-1207.eqiad.wmflabs tools.potd-feed lighttpd-potd-feed 261228
0 2015-12-28 13:50:13 tools-webgrid-lighttpd-1406.eqiad.wmflabs tools.pagecounts lighttpd-pagecounts 257473
0 2015-12-28 13:50:17 tools-webgrid-lighttpd-1410.eqiad.wmflabs tools.ios-crashes lighttpd-ios-crashes 257474
0 2015-12-28 13:50:21 tools-webgrid-lighttpd-1204.eqiad.wmflabs tools.awb lighttpd-awb 257475
0 2015-12-28 13:50:24 tools-webgrid-lighttpd-1401.eqiad.wmflabs tools.wikiradio lighttpd-wikiradio 425377
0 2015-12-28 13:50:27 tools-webgrid-lighttpd-1409.eqiad.wmflabs tools.dewkin lighttpd-dewkin 257476
0 2015-12-28 13:50:31 tools-webgrid-lighttpd-1404.eqiad.wmflabs tools.forrestbot lighttpd-forrestbot 257477
0 2015-12-28 13:50:35 tools-webgrid-lighttpd-1205.eqiad.wmflabs tools.authors lighttpd-authors 257478
0 2015-12-28 13:50:39 tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs tools.fronds lighttpd-fronds 257479
0 2015-12-28 13:50:43 tools-webgrid-lighttpd-1405.eqiad.wmflabs tools.giraffe lighttpd-giraffe 257480
0 2015-12-28 13:50:48 tools-webgrid-lighttpd-1408.eqiad.wmflabs tools.gorlingor lighttpd-gorlingor 257483
0 2015-12-28 13:50:53 tools-webgrid-lighttpd-1408.eqiad.wmflabs tools.alkamidbot lighttpd-alkamidbot 257484
0 2015-12-28 13:50:56 tools-webgrid-lighttpd-1401.eqiad.wmflabs tools.articleplaceholderwiki lighttpd-articleplaceholderwiki 257485
0 2015-12-28 13:51:00 tools-webgrid-lighttpd-1206.eqiad.wmflabs tools.wdq2sparql lighttpd-wdq2sparql 257486
0 2015-12-28 13:51:05 tools-webgrid-lighttpd-1201.eqiad.wmflabs tools.phetools lighttpd-phetools 479826
(...)
0 2015-12-28 14:15:42 tools-webgrid-lighttpd-1401.eqiad.wmflabs tools.wmcounter lighttpd-wmcounter 258246
```
which is when @Coren force-rescheduled all webservice jobs.
taking lighttpd-rxy as an example, we find:
```
2015-12-28T13:49:27.901690 No running webservice job found, attempting to start it
```
in service.log, and
```
2015-12-28 13:49:27: (server.c.1512) server stopped by UID = 0 PID = 32558
2015-12-28 13:49:30: (log.c.166) server started
2015-12-28 14:15:45: (log.c.166) server started
```
in error.log. PID 32558 of course doesn't exist anymore.
After restarting, the redis links were not updated correctly, although it's not immediately obvious from this why. Both tools-proxy-01's and tools-proxy-02's redis logs are empty (!) but that might be because things are working (the old log files only contain error situations)