Time to move on. We'll have tools-precise made available for people who still want to use precise.
This wouldn't affect anything else. tools-trusty will still point to a resolvable address.
We will stagger this so that there is always a bastion or two for people to connect from.
This should also provide two hosts, so when one is down due to any hardware failure, we can just switch the floating IP to the other host.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T143349 Deprecate precise instances in Labs by 2017-03-31 | |||
| Resolved | yuvipanda | T94790 Phase out precise instances from Tool Labs | |||
| Resolved | yuvipanda | T91863 Make tools-login / bastion hosts redundant and move them to trusty |
Actually, since they are just DNS entries, I can just point them to a different instance, and make backport ones for the older hosts. So people would just get routed to the new address at some point...
This would have people perhaps freaking out with the ssh key mismatch. Perhaps an motd for the occassion?
tools-login is now tools-bastion-01, and maybe I can create tools-bastion-02, and *that* is dev, and they share ssh certificates and we can switch them around if needed (for redundancy!).
After that we can get rid of tools-login and tools-dev and tools-trusty. And I guess we can actually not provide a precise instance at all.
Hmm, actually we probably need to provide a precise instance as long as we have tools running on precise.
I think since we moved the crontabs from tools-login, there is no real compelling reason to differentiate between different bastions; users should be nice to each other everywhere :-).
I still have a Bad Feeling™ about instances sharing host keys. The possibility of one host name being used for multiple instances (yes, you didn't propose that, but …) makes me even more uncomfortable as it makes debugging impossible. ("Did you encounter the problems on tools-dev or tools-dev?")
(Caution: Before tools-login is removed, the toolwatcher role must be enabled on another instance.)
ah, right. I wonder if toolwatcher should actually be on tools-master or something? I guess we can put that up on both tools-master and tools-shadow, so it has redundancy...
@scfc: So the benefits of having bastion-01 and bastion-02 is that when a virt node goes down we can just switch the floating IP as well (hmm, need to see if you can have two IPs refer to the same instance). Also, on -dev if you're compiling something with -j8 you can lock up the CPU for a fair bit... And I"d like to still provide users that ability :D
Re-using host keys also makes my skin crawl, but I still have no idea why....
I think toolwatcher could run on two instances with low risk of race conditions.
Regarding redundancy for bastions, sure, but I do remember vividly on Sunday wondering why ssh tools-login.eqiad.wmflabs connected me to tools-bastion-01.eqiad.wmflabs before I noticed my corresponding setting in ~/.ssh/config :-). Having to use another bastion instance is only relevant for active developers who want to log in during an outage of one virtual node, but not reading labs-l or #wikimedia-labs. So to me the benefit feels rather small (compared to redundancy for web services & Co.).
Alright, tools-dev should move in, say, a week. 22nd April.
Writing announcement email now.