As per https://wikitech.wikimedia.org/wiki/Incident_documentation/20150217-LabsOutage
Reliance on a single web proxy (both within tools and outside of tools) means that a partial labs outage reads to the outside world as a complete labs outage. We need some kind of failover or redundancy for this.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T90534 Make toolforge reliable enough (tracking) | |||
| Open | None | T91068 Set up a schedule for doing failover exercises for toollabs | |||
| Resolved | Andrew | T90542 Make sure that toollabs can function fully even with one virt* host fully down | |||
| Resolved | yuvipanda | T89995 ToolLabs web proxy tolerate the failure of virt host | |||
| Declined | yuvipanda | T91484 Monitor that the redundant webproxies have same state in terms of what they are proxying to whom |
It seems to me unlikely that the proxy can be made HA without a redesign, but having a warm standby ready to take over at the flip of a switch would probably be a sufficient recovery method (presuming, of course, that it lives on different hardware)
I think Labs is too intertwined that having the Labs/Tools proxy back up sooner than the rest is very useful. Fixing this hardware failure for the most part in less than three hours was pretty spectacular (thanks!) and (IMHO) very good for a less-than-production-grade environment.
The SPOF is probably more "North America, day time" :-). I'd prefer if the ops team could make sure that the know-how is also available on a Saturday night with the "usual suspects" partying, hiking, ill, etc., i. e. can another ops solve a similar situation with the existing training and documentation in, say, less than twice that time frame?
@Andrew, you are proposing to work on this task during the Wikimedia Hackathon in Lyon. Please consider associating this task with Wikimedia-Hackathon-2015.
Toollabs was out twice over the last few days because of non-redundancy in tools-webproxy
Change 193334 had a related patch set uploaded (by Yuvipanda):
tools: Make portgrabber also ping two additional webproxies
Change 193334 merged by Yuvipanda:
tools: Make portgrabber also ping two additional webproxies
Change 193343 had a related patch set uploaded (by Yuvipanda):
tools: Make uwsgi & nodejs services also ping additional proxies
Change 193343 merged by Yuvipanda:
tools: Make uwsgi & nodejs services also ping additional proxies
So we will eventually have two proxies - tools-webproxy-01 and tools-webproxy-02, and they'll be hotspares. Webservices will register with each of them when they are started. Switching between them would be just moving the floating IP from one to the other. Will add documentation soon.
Change 193345 had a related patch set uploaded (by Yuvipanda):
tools: Keep proxylistener socket open
Alright, so toollabs webproxy is now running on tools-webproxy-01, with a hotspare in tools-webproxy-02. To switch to the spare, go to Special:NovaAddress on wikitech, and reassociate the IP for tools.wmflabs.org to the other one. tools-webproxy, the old machine, is also kept on for now, but will be removed next week. The new machines are also trusty, rather than precise.
I'm not particularly sure where to document this..
https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin has documentation now :D
Tested the failover. Worked perfectly.
Haven't tested instructions on bringing back a dead instance, though.
My understanding is that dynamicproxy holds the proxying information essentially in memory, i. e. it starts with an empty plate. In that case, tools that start their webservice between start(tools-webproxy-01/dynamicproxy) and start(tools-webproxy-02/dynamicproxy) would lose their proxy when the floating IP is switched to tools-webproxy-02. This can be worked around by restarting all webservices that are not registered with the active proxy (which could be automatically monitored).
Should we just document this/set up monitoring, or do you want to implement some data sharing between the proxies?