see https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin#Webservers
I would happily fix many issues like occasional apache breakages etc, but it's really very hard because stuff is pretty much undocumented.
If Coren wouldn't document how tool labs work, he will probably remain the only 1 person who can ever fix anything there...
Version: unspecified
Severity: normal
URL: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin#Webservers
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T57946 Wikimedia Labs system admin (sysadmin) documentation sucks | |||
| Invalid | None | T96884 Document email setup |
I think this is fairly important, and we should make better docs now. New tools admins would be joining us shortly, and this page should be much better.
I've moved the old page to https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin/Archive and am creating a new page.
Tested the failover. Worked perfectly.
Haven't tested instructions on bringing back a dead instance, though.
Nothing is ever going to happen with this ticket as it's too broad to be useful. We are constantly trying to improve, and helping others improve the documentation, and thinking about the documentation of Tools.
Things are in a totally different state now than 2014.