Page MenuHomePhabricator
Create Task
Maniphest T341984

Update Kubernetes clusters to >1.25
Open, HighPublic
Actions

Description

Umbrella task to track the work required towards upgrading our Kubernetes clusters to Kubernetes >1.25 (1.24 is EOL on 2023-07-28).

We're currently running 1.23 which went EOL on 2023-02-08 and there are some bigger requirement to be dealt with before moving to a newer version:

  • We need to migrate away from docker ad container runtime: T269684
  • We need to migrate away from PodSecurityPolicies: T273507

Together with the Kubernetes update, we need to update the following other components:

  • Calico
  • Istio
  • cert-manager
  • kserve
  • knative-serving
  • coredns
  • helm

Preparation for the Kubernetes update

  • Ensure all our charts are compatible with the new Kubernetes version (currently validating against 1.27)
  • Read Kubernetes changelogs (yellow/red flags just linked below each version. Tick the box if all action required items have been addressed, use ✅ for single items), https://relnotes.k8s.io
  • v1.24
  • Action Required
  • Note
  • v1.25
  • Action Required
  • Note

Upgrade process

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJMeybohmT353233 Outage of wikikube codfw apiservers
 OpenNoneT341984 Update Kubernetes clusters to >1.25
 OpenJMeybohmT273507 PodSecurityPolicies will be deprecated with Kubernetes 1.21
 OpenJMeybohmT362978 Update all helm modules and charts to be compatible with the restricted PSS
 OpenNoneT269684 [EPIC] Docker deprecation as a container runtime enginer for kubernetes.
OpenNoneT362408 Migration to containerd and away from docker
 Resolved• ayounsiT306649 Agree strategy for Kubernetes BGP peering to top-of-rack switches
 ResolvedelukeyT308418 Add missing failure domain labels to ml-serve-* clusters
 ResolvedJMeybohmT277876 Reserve resources for system daemons on kubernetes nodes
 ResolvedClement_GoubertT346422 Move 10% of mediawiki external requests to mw on k8s
 OpenelukeyT277677 Write a cookbook to set a k8s cluster in maintenance mode
 ResolvedJMeybohmT260663 Create a cookbook for depooling one or all services from one kubernetes cluster
 ResolvedJMeybohmT324959 Scrape controller-manager and scheduler metrics
 OpenNoneT330060 etcd cluster reimage strategies to use with the K8s upgrade cookbook
 OpenNoneT336861 Fix naming confusion around main/wikikube kubernetes clusters
 OpenNoneT365571 Rename wikikube worker nodes during OS reimage
 DuplicateNoneT366085 Relabel kubernetes2032 to wikikube-worker2002
 DuplicateNoneT366468 Relabel kubernetes2023 to wikikube-worker2001
 ResolvedRequestJclark-ctrT366583 hw troubleshooting: firmware upgrade for mw1358.eqiad.wmnet
 OpenNoneT334234 Migrate to node-role.kubernetes.io/control-plane label/taint
 StalledhnowlanT353464 Migrate wikikube control planes to hardware nodes
 DuplicateClement_GoubertT348466 Rethink kubernetes etcd storage
 ResolvedJMeybohmT363307 Co-locate kube-apiserver and etcd on new staging control plane nodes
 ResolvedJMeybohmT363310 Site: codfw 1 VM request for staging-codfw kube-apiserver
 ResolvedJMeybohmT364740 Site: codfw 2 VM request for staging-codfw kube-apiserver
 ResolvedJMeybohmT364746 Site: eqiad 3 VM request for staging-eqiad kube-apiserver
 ResolvedRobHT365499 update idrac firmware on wikikube-ctrl1001
 OpenRequestPapaulT366205 codfw:(3) wikikube-ctrl NIC upgrade to 10G
 OpenRequestVRiley-WMFT366204 eqiad:(3) wikikube-ctrl NIC upgrade to 10G

Event Timeline

JMeybohm triaged this task as Medium priority.Jul 17 2023, 12:24 PM
JMeybohm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 17 2023, 12:24 PM
JMeybohm added subtasks: T273507: PodSecurityPolicies will be deprecated with Kubernetes 1.21, T269684: [EPIC] Docker deprecation as a container runtime enginer for kubernetes. , T306649: Agree strategy for Kubernetes BGP peering to top-of-rack switches, T277876: Reserve resources for system daemons on kubernetes nodes, T277677: Write a cookbook to set a k8s cluster in maintenance mode, T324959: Scrape controller-manager and scheduler metrics, T330060: etcd cluster reimage strategies to use with the K8s upgrade cookbook.Jul 17 2023, 12:26 PM
JMeybohm moved this task from Incoming 🐫 to ⎈Kubernetes on the serviceops board.
JMeybohm updated the task description. (Show Details)Jul 17 2023, 12:31 PM
JMeybohm added a subtask: T336861: Fix naming confusion around main/wikikube kubernetes clusters.Jul 17 2023, 12:33 PM
akosiaris subscribed.Jul 18 2023, 8:21 AM
JMeybohm changed the status of subtask T277876: Reserve resources for system daemons on kubernetes nodes from Open to Stalled.Aug 21 2023, 8:32 AM
fgiunchedi mentioned this in T320562: OpenSearch index provisioned for Jaeger .Sep 6 2023, 9:54 AM
JMeybohm added a subtask: T334234: Migrate to node-role.kubernetes.io/control-plane label/taint.Sep 20 2023, 4:57 PM
JMeybohm closed subtask T324959: Scrape controller-manager and scheduler metrics as Resolved.Sep 21 2023, 2:40 PM
JMeybohm raised the priority of this task from Medium to High.Sep 22 2023, 9:12 AM
JMeybohm closed subtask T277876: Reserve resources for system daemons on kubernetes nodes as Resolved.Sep 26 2023, 9:14 AM
• ayounsi closed subtask T306649: Agree strategy for Kubernetes BGP peering to top-of-rack switches as Resolved.Dec 12 2023, 9:54 AM
JMeybohm added a comment.EditedMar 4 2024, 8:52 AM

With the next k8s upgrade we already have the following dependency problems:

  • We need to migrate to containerd before moving to k8s >=1.24 (T269684)
  • containerd version (< 1.6) in bullseye is only supported in kubelet <=1.25 (see)
  • PSPs gone in >=1.25 (T273507)
  • VAPs available in >=1.26 (T273507)

With this in mind (and all the other components version requirements left out for now) we drafted the following update plans for the wikikube clusters. The main problem is the need for containerd >=1.6 if we want to go newer then kubelet 1.25 which would require us to reimage all nodes. In previous updates, that could easily be done during a cluster downtime, but with >130 nodes per cluster (and counting) that is probably no longer feasible (or at least requires extended cluster downtime and still puts us under some level of stress).

One big reimage plan
This is how we did it in the past. Downtime one cluster (during DC switchover) and update/reimage all of its nodes at once.

  • Switch to containerd (node per node)
  • Move everything that is not medawiki to PSSs (depending on how T273507 goes)
  • Upgrade to k8s 1.29, reimaging all nodes to bookworm (while cluster is depooled)
  • Add VAP to enforce policy to mediawiki namespaces (depending on how T273507 goes)

Rolling reimage plan
This idea leverages the fact that there is a supported version skew between kube-apiserver and kubelets.

  • Switch to containerd (node per node)
  • Move everything that is not medawiki to PSSs (depending on how [[PSP replacement]] goes)
  • Upgrade apiserver to 1.27, kubelets to 1.25 (still bullseye, containerd 1.4)
  • Add VAP to enforce policy to mediawiki namespaces (depending on how T273507 goes)
  • rolling reimage of nodes to bookworm and kubelet 1.27

Decouple reimage plan
This would require us to backport the bookworm containerd version (1.6) to bullseye.

  • Switch to containerd (node per node)
  • Move everything that is not medawiki to PSSs (depending on how T273507 goes)
  • Upgrade to k8s 1.29
  • Add VAP to enforce policy to mediawiki namespaces (depending on how T273507 goes)
  • Do bookworm reimaging whenever we like
JMeybohm added a subtask: T353464: Migrate wikikube control planes to hardware nodes.Mar 5 2024, 11:38 AM
JMeybohm added a subtask: T348466: Rethink kubernetes etcd storage.
akosiaris mentioned this in T362408: Migration to containerd and away from docker.Apr 12 2024, 1:44 PM
BTullis mentioned this in T362999: Decide on which postgresql operator to use.Mon, May 13, 11:16 AM
BTullis edited projects, added Data-Platform-SRE; removed Shared-Data-Infrastructure.Fri, May 17, 9:33 AM
BTullis moved this task from Incoming to Watching on the Data-Platform-SRE board.
jijiki changed the status of subtask T353464: Migrate wikikube control planes to hardware nodes from Open to Stalled.Thu, May 30, 10:50 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL