Page MenuHomePhabricator
Create Task
Maniphest T278238

Recreate ml-etcd2002 in a different row
Closed, ResolvedPublic
Actions

Description

As Moritz noticed, we have two ml-etcd nodes on the same ganeti host:

elukey@ganeti2021:~$ sudo gnt-instance list | grep ml-etcd
ml-etcd2001.codfw.wmnet         kvm        debootstrap+default ganeti2019.codfw.wmnet running      3.0G
ml-etcd2002.codfw.wmnet         kvm        debootstrap+default ganeti2019.codfw.wmnet running      3.0G
ml-etcd2003.codfw.wmnet         kvm        debootstrap+default ganeti2015.codfw.wmnet running      3.0G

This is my bad since I have created the cluster without double checking. If possible I'd need to destroy / re-create ml-etcd2002 on a different ganeti row/host.

Details

SubjectRepoBranchLines +/-
install_server: change mac address for ml-etcd2002operations/puppetproduction+1 -1
Remove ml-etcd2002 SRV record for decomoperations/dnsmaster+0 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT272917 Lift Wing proof of concept
 ResolvedklausmanT272918 Create ml-serve k8s cluster
 ResolvedelukeyT278238 Recreate ml-etcd2002 in a different row

Event Timeline

elukey created this task.Mar 23 2021, 2:39 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMar 23 2021, 2:39 PM
elukey moved this task from Unsorted to Backlog/SRE on the Machine-Learning-Team board.Mar 24 2021, 6:58 AM
elukey@ml-etcd2002:~$ etcdctl -C https://ml-etcd2002.codfw.wmnet:2379 cluster-health
member 367f7076aea55538 is healthy: got healthy result from https://ml-etcd2002.codfw.wmnet:2379
member 3eaef5f31c9d4f07 is healthy: got healthy result from https://ml-etcd2001.codfw.wmnet:2379
member 6ec81f119df22c02 is healthy: got healthy result from https://ml-etcd2003.codfw.wmnet:2379
cluster is healthy
elukey@ml-etcd2002:~$ curl -k -L https://ml-etcd2002.codfw.wmnet:2379/v2/stats/leader
{"message":"not current leader"}
ops-monitoring-bot added a comment.Mar 24 2021, 7:20 AM

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: ml-etcd2002.codfw.wmnet

  • ml-etcd2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Mar 24 2021, 7:22 AM

Change 674486 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Remove ml-etcd2002 SRV record for decom

https://gerrit.wikimedia.org/r/674486

gerritbot added a project: Patch-For-Review.Mar 24 2021, 7:22 AM

Change 674486 merged by Elukey:
[operations/dns@master] Remove ml-etcd2002 SRV record for decom

https://gerrit.wikimedia.org/r/674486

elukey renamed this task from Recreate ml-etcd2001 in a different row to Recreate ml-etcd2002 in a different row.Mar 24 2021, 8:01 AM
elukey updated the task description. (Show Details)
gerritbot added a comment.Mar 24 2021, 8:01 AM

Change 674523 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] install_server: change mac address for ml-etcd2002

https://gerrit.wikimedia.org/r/674523

gerritbot added a comment.Mar 24 2021, 8:02 AM

Change 674523 merged by Elukey:
[operations/puppet@production] install_server: change mac address for ml-etcd2002

https://gerrit.wikimedia.org/r/674523

elukey added a comment.Mar 24 2021, 10:50 AM

After a little battle I was able to remove/add the etcd node:

elukey@ml-etcd2001:~$ sudo etcdctl -C https://ml-etcd2001.codfw.wmnet:2379 cluster-health
member 3eaef5f31c9d4f07 is healthy: got healthy result from https://ml-etcd2001.codfw.wmnet:2379
member 6ec81f119df22c02 is healthy: got healthy result from https://ml-etcd2003.codfw.wmnet:2379
member 8ac1f758490b9cfc is healthy: got healthy result from https://ml-etcd2002.codfw.wmnet:2379
cluster is healthy

Also:

elukey@ganeti2021:~$ sudo gnt-instance list | grep ml-etcd
ml-etcd2001.codfw.wmnet         kvm        debootstrap+default ganeti2019.codfw.wmnet running      3.0G
ml-etcd2002.codfw.wmnet         kvm        debootstrap+default ganeti2014.codfw.wmnet running      3.0G
ml-etcd2003.codfw.wmnet         kvm        debootstrap+default ganeti2015.codfw.wmnet running      3.0G
elukey closed this task as Resolved.Mar 24 2021, 10:50 AM
klausman added a comment.Apr 6 2021, 7:50 AM

For the record: I just checked, and no, it wasn't you who created the VM on the wrong Ganeti host, but me. Thanks for fixing it!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits