I began this process at this comment T215531#5363490
That's a fine way to bury useful information, so this task is to produce and use a document from that experience and ensure that our clusters are off to a good start at 1.15.5 now that we have the packages for it thanks to T236824: Toolforge: new k8s: get new deb packages for 1.15.4 or 1.15.5.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | • Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
| Restricted Task | |||||
| Resolved | bd808 | T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail | |||
| Resolved | • Bstorm | T236565 "tools" Cloud VPS project jessie deprecation | |||
| Resolved | aborrero | T101651 Set up toolsbeta more fully to help make testing easier | |||
| Resolved | • Bstorm | T166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf) | |||
| Resolved | • Bstorm | T246059 Add admin account creation to maintain-kubeusers | |||
| Resolved | • Bstorm | T154504 Make webservice backend default to kubernetes | |||
| Declined | None | T245230 Investigate cpu/ram requests and limits for DaemonSets pods | |||
| Resolved | • Bstorm | T214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster | |||
| Resolved | • Bstorm | T237789 Document (and execute) the upgrade process for the new Toolforge K8s cluster |
The upgrade cycle is meant to be done on a six month process to stay ahead of certificate refreshes. CVEs could easily have us doing it more often than that. We should expect to be upgrading fairly often.
@aborrero upgraded the clusters. I'm adding the doc here: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Upgrading_Kubernetes
I added comments in the wikitext to drain the control plane nodes as you upgrade. This is recommended in the upgrade to 1.16, so we'll want to uncomment that when we do that.