Page MenuHomePhabricator
Create Task
Maniphest T344230

Get aux-k8s cluster row-redundant and with more workers
Closed, ResolvedPublic
Actions

Description

Currently all cluster componentes (etcd, control planes, workers) are in the same ganeti group which is very suboptimal:

$ sudo gnt-instance list -o name,pnode.group | grep aux
aux-k8s-ctrl1001.eqiad.wmnet        A
aux-k8s-ctrl1002.eqiad.wmnet        A
aux-k8s-etcd1001.eqiad.wmnet        A
aux-k8s-etcd1002.eqiad.wmnet        A
aux-k8s-etcd1003.eqiad.wmnet        A
aux-k8s-worker1001.eqiad.wmnet      A
aux-k8s-worker1002.eqiad.wmnet      A

This obviously affects availability of the cluster and can affect (and has!) deployments with podAntiAffinity rules requiring row-diverse pod placement (like calico-typha).

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Aug 15 2023, 9:34 AM
fgiunchedi renamed this task from Get aux-k8s cluster row-redundant to Get aux-k8s cluster row-redundant and with more workers.Aug 15 2023, 9:37 AM
fgiunchedi updated the task description. (Show Details)
gerritbot added a comment.Aug 15 2023, 9:58 AM

Change 949002 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Remove podAntiAffinity for calico-typha on aux

https://gerrit.wikimedia.org/r/949002

gerritbot added a project: Patch-For-Review.Aug 15 2023, 9:58 AM
gerritbot added a comment.Aug 15 2023, 12:35 PM

Change 949002 merged by Filippo Giunchedi:

[operations/deployment-charts@master] Remove podAntiAffinity for calico-typha on aux

https://gerrit.wikimedia.org/r/949002

Maintenance_bot removed a project: Patch-For-Review.Aug 15 2023, 1:10 PM
JMeybohm subscribed.Aug 15 2023, 1:26 PM

To be able to deploy calico changes we dropped the anti affinity rule from the typha deployment (T333302) this should be undone when the aux cluster is row redundant.

JMeybohm updated the task description. (Show Details)Aug 15 2023, 1:33 PM
JMeybohm added a project: Infrastructure-Foundations.Aug 31 2023, 1:14 PM
JMeybohm added a subscriber: akosiaris.
• lmata moved this task from Inbox to Backlog on the Observability-Tracing board.Sep 25 2023, 5:28 PM
• joanna_borun assigned this task to CDanis.Jan 22 2024, 4:18 PM
• joanna_borun triaged this task as Low priority.
elukey subscribed.Jul 9 2024, 8:33 AM

Came across this task with working on some K8s tasks. I know this is discouraged, but could we try https://wikitech.wikimedia.org/wiki/Ganeti#Renumber_(aka_change_network)_a_VM with one VM (maybe one of the control plane nodes or one of the etcd ones) and see how it goes?

elukey added a project: User-Elukey.Sep 2 2024, 3:41 PM
elukey added a comment.Sep 2 2024, 3:46 PM

Proposal for a first step, that doesn't involve renumbering:

  1. Create aux-k8s-ctrl1003.eqiad.wmnet (same config as the other two ctrl nodes).
  2. Create aux-k8s-worker1003.eqiad.wmnet (same config as the other two worker nodes).
  3. Add both of them to the cluster, and make sure everything works as expected.
  4. Drop aux-k8s-ctrl1001.eqiad.wmnet and aux-k8s-worker1001.eqiad.wmnet from the k8s config, and delete any trace of those VMs.

We should be able to unblock the typha adffinity rules and have a better redundancy for the cluster. The etcd cluster may stay as it is, to be re-created when we'll upgrade to a newer version of k8s. Thoughts?

elukey moved this task from Backlog to Waiting for others on the User-Elukey board.Sep 4 2024, 9:42 AM
elukey added a comment.Sep 26 2024, 3:23 PM

Creating new VMs in T375746

elukey closed subtask T375746: eqiad: 2 VMs for k8s AUX as Resolved.Sep 30 2024, 9:05 AM
gerritbot added a comment.Sep 30 2024, 9:09 AM

Change #1076679 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add aux-k8s-{ctrl,worker}1003 to AUX K8s

https://gerrit.wikimedia.org/r/1076679

gerritbot added a project: Patch-For-Review.Sep 30 2024, 9:09 AM
gerritbot added a comment.Sep 30 2024, 9:12 AM

Change #1076681 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] Add aux-k8s-ctrl1003 to admin_ng's config for AUX

https://gerrit.wikimedia.org/r/1076681

gerritbot added a comment.Oct 1 2024, 9:38 AM

Change #1076681 merged by Elukey:

[operations/deployment-charts@master] admin_ng: drop unused aux-k8s control plane IPs

https://gerrit.wikimedia.org/r/1076681

gerritbot added a comment.Oct 1 2024, 1:13 PM

Change #1076679 merged by Elukey:

[operations/puppet@production] Add aux-k8s-{ctrl,worker}1003 to AUX K8s

https://gerrit.wikimedia.org/r/1076679

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2024, 1:31 PM
elukey closed this task as Resolved.Oct 1 2024, 3:10 PM

New status:

elukey@ganeti1028:~$ sudo gnt-instance list -o name,pnode.group | grep aux
aux-k8s-ctrl1001.eqiad.wmnet        A
aux-k8s-ctrl1002.eqiad.wmnet        A
aux-k8s-ctrl1003.eqiad.wmnet        D
aux-k8s-etcd1001.eqiad.wmnet        A
aux-k8s-etcd1002.eqiad.wmnet        A
aux-k8s-etcd1003.eqiad.wmnet        A
aux-k8s-worker1001.eqiad.wmnet      A
aux-k8s-worker1002.eqiad.wmnet      A
aux-k8s-worker1003.eqiad.wmnet      B

@CDanis I'd remove aux-k8s-ctrl1001.eqiad.wmnet and aux-k8s-worker1001.eqiad.wmnet if you are ok, and then possibly try to figure out how to add 2 more etcd vms (to expand the cluster to 5 and then drop aux-k8s-etcd1001.eqiad.wmnet and aux-k8s-etcd1002.eqiad.wmnet for example). Does it make sense? Do we also want to add more workers?

elukey reopened this task as Open.Oct 1 2024, 3:10 PM
ops-monitoring-bot added a comment.Oct 2 2024, 8:55 AM

depool host aux-k8s-worker1001.eqiad.wmnet by elukey@cumin1002 with reason: None

ops-monitoring-bot added a comment.Oct 2 2024, 8:57 AM

Cookbook cookbooks.sre.k8s.pool-depool-node started by elukey@cumin1002 depool for host aux-k8s-worker1001.eqiad.wmnet completed:

  • aux-k8s-worker1001.eqiad.wmnet (PASS)
    • Host aux-k8s-worker1001.eqiad.wmnet depooled from aux-k8s-eqiad
ops-monitoring-bot added a comment.Oct 2 2024, 8:57 AM

depool host aux-k8s-ctrl1001.eqiad.wmnet by elukey@cumin1002 with reason: None

ops-monitoring-bot added a comment.Oct 2 2024, 8:57 AM

Cookbook cookbooks.sre.k8s.pool-depool-node started by elukey@cumin1002 depool for host aux-k8s-ctrl1001.eqiad.wmnet completed:

  • aux-k8s-ctrl1001.eqiad.wmnet (PASS)
    • Host aux-k8s-ctrl1001.eqiad.wmnet depooled from aux-k8s-eqiad
elukey added a comment.Oct 2 2024, 8:58 AM

Drained the 1001 nodes in row A, current status:

root@deploy2002:~# kubectl get nodes
NAME                             STATUS                     ROLES           AGE    VERSION
aux-k8s-ctrl1001.eqiad.wmnet     Ready,SchedulingDisabled   control-plane   592d   v1.23.14
aux-k8s-ctrl1002.eqiad.wmnet     Ready                      control-plane   592d   v1.23.14
aux-k8s-ctrl1003.eqiad.wmnet     Ready                      control-plane   19h    v1.23.14
aux-k8s-worker1001.eqiad.wmnet   Ready,SchedulingDisabled   <none>          592d   v1.23.14
aux-k8s-worker1002.eqiad.wmnet   Ready                      <none>          592d   v1.23.14
aux-k8s-worker1003.eqiad.wmnet   Ready                      <none>          19h    v1.23.14

If nothing pops up I'll just decom those VMs tomorrow.

elukey closed subtask T376253: eqiad: 2 VMs for k8s etcd AUX as Resolved.Oct 2 2024, 12:46 PM
elukey added a comment.Oct 2 2024, 12:50 PM

For etcd, I think that we can just add one node at the time via https://wikitech.wikimedia.org/wiki/Etcd#Adding_a_new_member_to_the_cluster, make sure the cluster-health is good, and then drop aux-k8s-etcd100[1,2] via the delete member API.

gerritbot added a comment.Oct 11 2024, 2:54 PM

Change #1079534 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add aux-k8s-etcd1004 in service

https://gerrit.wikimedia.org/r/1079534

gerritbot added a project: Patch-For-Review.Oct 11 2024, 2:54 PM

Change #1079535 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add aux-k8s-etcd1005 in service

https://gerrit.wikimedia.org/r/1079535

gerritbot added a comment.Oct 11 2024, 2:58 PM

Change #1079539 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Add aux-k8s-etcd1004 to the aux-k8s SRV records

https://gerrit.wikimedia.org/r/1079539

gerritbot added a comment.Oct 11 2024, 2:58 PM

Change #1079540 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Add aux-k8s-etcd1005 to the Aux k8s SRV records

https://gerrit.wikimedia.org/r/1079540

elukey added a comment.Oct 11 2024, 3:00 PM

Procedure to expand etcd from 3 to 5:

At this point we should have a 5 nodes cluster, fully working and healthy. If all good, we'll be able to drop 1001 and 1002 via https://wikitech.wikimedia.org/wiki/Etcd#Removing_a_member_from_the_cluster

JMeybohm added a comment.Oct 14 2024, 7:39 AM

I've tried to add some documentation on how to add/remove control planes to https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remove_control-planes please extend if you feel like there is anything missing there ❤️

gerritbot added a comment.Oct 14 2024, 11:44 AM

Change #1079534 merged by Elukey:

[operations/puppet@production] Add aux-k8s-etcd1004 in service

https://gerrit.wikimedia.org/r/1079534

gerritbot added a comment.Oct 14 2024, 11:51 AM

Change #1079539 merged by Elukey:

[operations/dns@master] Add aux-k8s-etcd1004 to the aux-k8s SRV records

https://gerrit.wikimedia.org/r/1079539

gerritbot added a comment.Oct 14 2024, 12:01 PM

Change #1079540 merged by Elukey:

[operations/dns@master] Add aux-k8s-etcd1005 to the Aux k8s SRV records

https://gerrit.wikimedia.org/r/1079540

gerritbot added a comment.Oct 14 2024, 12:01 PM

Change #1079535 merged by Elukey:

[operations/puppet@production] Add aux-k8s-etcd1005 in service

https://gerrit.wikimedia.org/r/1079535

Stashbot added a comment.Oct 14 2024, 12:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-14T12:09:48Z] <elukey> increase etcd k8s aux cluster from 3 -> 5 - T344230

gerritbot added a comment.Oct 14 2024, 12:22 PM

Change #1079999 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Remove aux-k8x-{ctrl,worker}1001 from production

https://gerrit.wikimedia.org/r/1079999

ops-monitoring-bot added a comment.Oct 14 2024, 12:32 PM

cookbooks.sre.hosts.decommission executed by elukey@cumin1002 for hosts: aux-k8s-ctrl1001.eqiad.wmnet

  • aux-k8s-ctrl1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
ops-monitoring-bot added a comment.Oct 14 2024, 12:43 PM

cookbooks.sre.hosts.decommission executed by elukey@cumin1002 for hosts: aux-k8s-worker1001.eqiad.wmnet

  • aux-k8s-worker1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
gerritbot added a comment.Oct 14 2024, 12:45 PM

Change #1080011 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] kubernetes: change the AUX etcd urls nodes

https://gerrit.wikimedia.org/r/1080011

gerritbot added a comment.Oct 14 2024, 12:48 PM

Change #1079999 merged by Elukey:

[operations/puppet@production] Remove aux-k8x-{ctrl,worker}1001 from production

https://gerrit.wikimedia.org/r/1079999

gerritbot added a comment.Oct 14 2024, 12:49 PM

Change #1080011 merged by Elukey:

[operations/puppet@production] kubernetes: change the AUX etcd urls nodes

https://gerrit.wikimedia.org/r/1080011

elukey added a comment.Oct 14 2024, 12:51 PM

Next steps:

  • Reduce the etcd cluster from 5 to 3
  • Remove the following
In T344230#9093138, @JMeybohm wrote:

To be able to deploy calico changes we dropped the anti affinity rule from the typha deployment (T333302) this should be undone when the aux cluster is row redundant.

gerritbot added a comment.Oct 14 2024, 12:57 PM

Change #1080016 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Remove aux-k8s-etcd100[1,2] from the AUX client SRV records

https://gerrit.wikimedia.org/r/1080016

gerritbot added a comment.Oct 14 2024, 12:57 PM

Change #1080017 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Remove aux-k8s-etcd1001 from the AUX cluster's SRV records

https://gerrit.wikimedia.org/r/1080017

gerritbot added a comment.Oct 14 2024, 12:57 PM

Change #1080018 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Remove aux-k8s-etcd1002 from the AUX cluster's SRV records

https://gerrit.wikimedia.org/r/1080018

gerritbot added a comment.Oct 14 2024, 1:06 PM

Change #1080016 merged by Elukey:

[operations/dns@master] Remove aux-k8s-etcd100[1,2] from the AUX client SRV records

https://gerrit.wikimedia.org/r/1080016

gerritbot added a comment.Oct 14 2024, 1:12 PM

Change #1080017 merged by Elukey:

[operations/dns@master] Remove aux-k8s-etcd1001 from the AUX cluster's SRV records

https://gerrit.wikimedia.org/r/1080017

gerritbot added a comment.Oct 14 2024, 1:14 PM

Change #1080018 merged by Elukey:

[operations/dns@master] Remove aux-k8s-etcd1002 from the AUX cluster's SRV records

https://gerrit.wikimedia.org/r/1080018

gerritbot added a comment.Oct 14 2024, 1:20 PM

Change #1080022 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Remove aux-k8s-etcd100[1,2] from production

https://gerrit.wikimedia.org/r/1080022

ops-monitoring-bot added a comment.Oct 14 2024, 1:26 PM

cookbooks.sre.hosts.decommission executed by elukey@cumin1002 for hosts: aux-k8s-etcd1001.eqiad.wmnet

  • aux-k8s-etcd1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
ops-monitoring-bot added a comment.Oct 14 2024, 1:35 PM

cookbooks.sre.hosts.decommission executed by elukey@cumin1002 for hosts: aux-k8s-etcd1002.eqiad.wmnet

  • aux-k8s-etcd1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
elukey added a comment.Oct 14 2024, 2:49 PM

Way better now!

elukey@ganeti1028:~$ sudo gnt-instance list -o name,pnode.group | grep aux
aux-k8s-ctrl1002.eqiad.wmnet        A
aux-k8s-ctrl1003.eqiad.wmnet        D
aux-k8s-etcd1003.eqiad.wmnet        A
aux-k8s-etcd1004.eqiad.wmnet        D
aux-k8s-etcd1005.eqiad.wmnet        C
aux-k8s-worker1002.eqiad.wmnet      A
aux-k8s-worker1003.eqiad.wmnet      B
gerritbot added a comment.Oct 14 2024, 3:03 PM

Change #1080022 merged by Elukey:

[operations/puppet@production] Remove aux-k8s-etcd100[1,2] from production

https://gerrit.wikimedia.org/r/1080022

elukey closed this task as Resolved.Oct 14 2024, 3:06 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 14 2024, 3:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits