Page MenuHomePhabricator
Create Task
Maniphest T333929

toolforge: Move k8s control plane nodes to larger instance flavors
Closed, ResolvedPublic
Actions

Description

While investigating T333922: toolforge k8s control plane freezing and other stability issues we noticed that the Toolforge Kubernetes control plane nodes are undersized for the task as they're using the g2.cores2.ram4.disk40 flavor. We need to migrate those instances to a larger flavor.

Details

ProjectBranchLines +/-Subject
cloud/wmcs-cookbooksmain+3 -3wmcs_libs: inventory: refresh tools k8s control nodes
cloud/wmcs-cookbooksmain+23 -5toolforge: update firewall rules on etcd nodes when needed
cloud/wmcs-cookbooksmain+151 -60cookbooks: Generalize Toolforge add node cookbook to add control nodes
Customize query in gerrit

Related Objects
Search...

Event Timeline

taavi triaged this task as High priority.Apr 4 2023, 10:25 AM
taavi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2023, 10:25 AM
taavi added a parent task: T333922: toolforge k8s control plane freezing and other stability issues.Apr 4 2023, 10:27 AM
aborrero moved this task from Triage to Backlog on the Toolforge board.Apr 4 2023, 11:57 AM
aborrero moved this task from Inbox to Soon! on the cloud-services-team board.
taavi claimed this task.Apr 4 2023, 5:01 PM
taavi added a comment.Apr 4 2023, 5:07 PM

I tried but looks like we can't resize those instances as the g2 flavors are now unavailable, so I'm just going to re-create them with newer flavors.

Stashbot added a comment.Apr 4 2023, 6:45 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-04T18:45:05Z] <wm-bot2> Adding a new k8s CONTROL node (T333929) - cookbook ran by taavi@runko

Stashbot added a comment.Apr 4 2023, 6:46 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-04T18:46:35Z] <wm-bot2> Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

Stashbot added a comment.Apr 4 2023, 7:00 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-04T19:00:20Z] <wm-bot2> Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

taavi added a comment.Apr 4 2023, 7:27 PM

Added two more control nodes to the cluster, I'll wait until tomorrow before removing the old nodes and adding one extra new node.

gerritbot added a comment.Apr 4 2023, 7:28 PM

Change 905707 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] cookbooks: Generalize Toolforge add node cookbook to add control nodes

https://gerrit.wikimedia.org/r/905707

gerritbot added a project: Patch-For-Review.Apr 4 2023, 7:28 PM
gerritbot added a comment.Apr 5 2023, 7:37 AM

Change 905707 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] cookbooks: Generalize Toolforge add node cookbook to add control nodes

https://gerrit.wikimedia.org/r/905707

taavi mentioned this in rCCKB69f1ccbd2a96: cookbooks: Generalize Toolforge add node cookbook to add control nodes.Apr 5 2023, 7:38 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 5 2023, 8:10 AM
Stashbot added a comment.Apr 5 2023, 10:16 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-05T10:16:30Z] <wm-bot2> Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

Stashbot added a comment.Apr 5 2023, 10:41 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-05T10:41:14Z] <wm-bot2> Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

Stashbot added a comment.Apr 5 2023, 11:09 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-05T11:08:57Z] <wm-bot2> Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

Stashbot added a comment.Apr 5 2023, 11:21 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-05T11:21:52Z] <wm-bot2> Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

Stashbot added a comment.Apr 5 2023, 11:39 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-05T11:39:06Z] <wm-bot2> Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

gerritbot added a comment.Apr 5 2023, 12:09 PM

Change 905997 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] toolforge: update firewall rules on etcd nodes when needed

https://gerrit.wikimedia.org/r/905997

gerritbot added a project: Patch-For-Review.Apr 5 2023, 12:09 PM
taavi added a comment.Apr 5 2023, 12:43 PM

Added one more node and removed -control-1. I'll again wait some time before continuing to remove the last old nodes.

gerritbot added a comment.Apr 5 2023, 1:33 PM

Change 906022 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] wmcs_libs: inventory: refresh tools k8s worker nodes

https://gerrit.wikimedia.org/r/906022

gerritbot added a comment.Apr 5 2023, 2:15 PM

Change 905997 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] toolforge: update firewall rules on etcd nodes when needed

https://gerrit.wikimedia.org/r/905997

taavi mentioned this in rCCKB5920c1b9d789: toolforge: update firewall rules on etcd nodes when needed.Apr 5 2023, 2:16 PM
gerritbot added a comment.Apr 5 2023, 2:26 PM

Change 906022 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] wmcs_libs: inventory: refresh tools k8s control nodes

https://gerrit.wikimedia.org/r/906022

taavi mentioned this in rCCKBed9a9e65365e: wmcs_libs: inventory: refresh tools k8s control nodes.Apr 5 2023, 2:26 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 5 2023, 2:30 PM
Stashbot added a comment.Apr 7 2023, 2:34 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-04-07T14:34:31Z] <wm-bot2> drained, depooled and removed k8s control node tools-k8s-control-3 (T333929) - cookbook ran by taavi@runko

taavi closed this task as Resolved.Apr 7 2023, 2:35 PM
taavi mentioned this in T333059: Spread Toolforge tools to multiple Kubernetes clusters.Apr 7 2023, 7:49 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL