| • toan | |
| Jun 23 2022, 8:38 AM |
| F35453758: image.png | |
| Aug 15 2022, 12:40 PM |
We are currently using version 1.21.x, a version set for retirement/EOL the 28th this month
We should upgrade this to run on a newer version locally and for our staging/production environments
This would involve resolving the deprecation of some of our v1beta usage, mostly ingresses it seems but there could be a lot of hidden gems/problems that would appear once we start doing this.
AC
Useful links:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Evelien_WMDE | T311205 🔷 Upgrade kubernetes from 1.21 to 1.22 | |||
| Resolved | Evelien_WMDE | T322542 🔷 Remove all instances of removed 1.22 k8s APIs | |||
| Resolved | Evelien_WMDE | T322543 🔵 Remove beta ingress API from wikibase-ingress chart | |||
| Resolved | Evelien_WMDE | T311505 🔵 Resolve Kubernetes client bump from dependabot PR | |||
| Resolved | Evelien_WMDE | T322552 🔵 Update nginx-ingress to ingress-nginx chart v4.2.5 |
See https://cloud.google.com/kubernetes-engine/docs/release-notes for possible targets for us to aim for. I would suggest we aim for 1.22 as the Regular version
I looks like to more forwards to 1.22 on GKE we are only blocked by two APIs we call that will no longer be around:
| API | User agent | Total calls (last 30 days) | Last called |
| /apis/networking.k8s.io/v1beta1/ingresses | nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format | 12698 | 14 Aug 2022, 05:04:00 |
| /apis/extensions/v1beta1/ingresses | Go-http-client/2.0 | 18 | 8 Aug 2022, 15:54:00 |
Looks to me like the only real thing required would be to update the version of the nginx-ingress charts. This however probably does result in us needing to remove the "pinned" bitnami charts repository
note that v1.22 reached EOL as of today https://kubernetes.io/releases/#release-v1-22
I refactored the UI chart in this PR to use the new stable ingress API. more details here: https://github.com/wbstack/charts/pull/104
I created similar PRs for the API, QueryService, and QueryService UI charts as well
@Rosalie_WMDE and I had a chat. Right now we want to at a minimum upgrade to 1.22 since this is supported by GKE (https://cloud.google.com/kubernetes-engine/docs/release-notes#current_versions). GKE also supports 1.23 and 1.24 but not 1.25 yet. We probably want to move up the versions incrementally because some services that we use (ingress-nginx/nginx-ingress for example) don't support a wide enough version range for us to make the jump in one step.
Today we upgraded the staging clusters wbaas-2 control plane and node-pools to 1.22.15-gke.1000. This was a 3-step process (control plane, oldest node-pool medium-pool, the other node-pool) and was done via the Google Cloud Console UI (docs).
Note: In the future, if we activate auto-upgrade for the node pools, upgrading just the control plane will be enough.
During the upgrade the following alerts fired:
Throughout the upgrade of the second node pool (standard-pool, 4 nodes) there popped up a message in the Cloud Console indicating that the upgrades are a bit delayed, possibly because of pod disruption budgets or grace periods.
The current upgrade delay of 7 minutes per node indicates your Pod Disruption Budgets may need attention. Learn more
Everything was completed after roundabout ~1h20m
The upgrade on wbaas-3 was concluded today as well, pretty much the same experience as with staging yesterday.
What was noticeable is that some node upgrades were delayed again, this time I spotted the reason: missing set of elastic search availability - it waited because only 1 pod was allowed to be unavailable at any time, and currently the time until one of them reports as ready is around ~40m after pod start.
The production cluster is now running on 1.22.15-gke.1000