Etcd is actually installed on VMs in eqiad, we want to move it/add to it 3 servers in different rack rows.
Those are going to be the 3 eqiad zookeeper hosts.
What we need is:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Joe | T97029 integrate (pybal|varnish)->varnish backend config/state with etcd or similar | |||
| Resolved | Joe | T101713 Install etcd in multiple rows/racks |
@Joe, can I work on this next week when I am no longer in the office, or do you need to get it done sooner?
I made a little migration plan here:
http://etherpad.wikimedia.org/p/Jessie-Zookeeper
I think I can handle the new IP assignment as part of the reinstall. We will need coordination for network switch ACL and port VLAN changes.
TBD: Should we remove the $ANALYTICS_NETWORKS ferm restriction on zookeeper ports?
https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/zookeeper.pp#L81
@Cmjohnson FYI, we will be renaming analytlics1023,analytics1024,analytics1025 to conf1001,conf1002,conf1003 respectively.
Change 218634 had a related patch set uploaded (by Ottomata):
Prep for reinstall and rename of analytics zookeeper hosts
Change 218639 had a related patch set uploaded (by Ottomata):
New IPs for conf100[[123]
Ok, got a partman recipe working (I think). I had to merge in order to test with d-i-test. @Joe could you post review it?
I *think* it is working, although I do get an error on d-i-test near the end of the install process:
Unable to install GRUB in /dev/sda. Executing 'grub-install /dev/sda' failed. This is a fatal error.
Not sure if that is just an artifact of the fact that d-i-test is virtual and has /dev/vda. Or...do I need a /boot partition?
Change 218634 merged by Ottomata:
Prep for reinstall and rename of analytics zookeeper hosts
Change 218991 had a related patch set uploaded (by Ottomata):
analytics1023 -> conf1001
Status report!
conf1001 has been installed and is running zookeeper. I will do conf1002 and conf1003 tomorrow.
DONE! conf1001-conf1003 are now Zookeeper hosts running Jessie. I hand them to you!
I have a few more post-migration tasks to do (documentation, Jessie jmxtrans package, etc.), but nothing that affects production operation.
Change 219379 had a related patch set uploaded (by Ottomata):
Remove site.pp references to analytics102[345] They have been renamed conf100[123]
Change 219379 merged by Ottomata:
Remove site.pp references to analytics102[345] They have been renamed conf100[123]
Change 219380 had a related patch set uploaded (by Ottomata):
Rename analytics102[345] mgmt entries to conf100[123]
Change 219380 merged by Ottomata:
Rename analytics102[345] mgmt entries to conf100[123]
FYI: I have moved the Analytics/Cluster/Zookeeper/* pages to top level paths:
https://wikitech.wikimedia.org/wiki/Zookeeper
https://wikitech.wikimedia.org/wiki/Zookeeper/Administration
I added conf1001 to the cluster this morning. I will add the remainder in the next couple of hours.
I added conf1002 and conf1003 too, and removed etcd1003 so that now the ganeti-based hosts don't have the quorum by themselves.
I may remove them alltoghether at a later point in time