We're planning to update the wikikube codfw cluster to kubernetes 1.31 on Monday, 2025-06-23 during the [[ https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250623T1000 | UTC mid-day MW-Infra window ]], 10:00 - 11:00 UTC (which gives us another 2 hours before the UTC afternoon backport window).
Required patches:
- Patches similar to what was required for staging-eqiad: {T389045}
- The new, bigger, Pod IP pools can be found at {T375845}, routers and ToR switches have already been configured with the new ranges
- Plus changing the kubernetesVersion for mw deployments to codfw because {T388390} / {T388969}
Since we're going to depool the whole codfw cluster we will be running a test depool during the UTC mid-day MW-Infra window on 2025-06-18.
As of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1127859 we're still running mw-web and mw-api-ext with replicas suitable for single-DC serving. So for the depool test, no further changes are required.
Upgrade process is:
- `scap lock --all "Kubernetes upgrade"`
- `cookbook sre.k8s.pool-depool-cluster depool codfw codfw`
- double check all services are depooled `cookbook sre.k8s.pool-depool-cluster status codfw codfw`
- Take a note on which services are currently deployed (`helm list -A`)
- `cookbook sre.k8s.wipe-cluster --k8s-cluster wikikube-codfw -H 2 --reason "Kubernetes upgrade"`
- Merge patches after "//Cluster's state has been wiped. //"
- Deploy admin_ng
- deploy istio CRDs first and delete namespace (so that it can be recreated by helm)
- `istioctl-1.24.2 install --set profile=remote --skip-confirmation && kubectl delete ns istio-system`
- helmfile sync admin_ng
- `istioctl-X.X manifest apply -f /srv/deployment-charts/custom_deploy.d/istio/<your-cluster>/config.yaml`
- Deploy all the services
- `deploy_all.sh`
- Deploy mediawiki: `scap sync-world --k8s-only -Dbuild_mw_container_image:False`
- Apply updated network policies to other clusters
- repool