To get ready to perform upgrades to Toolforge Kubernetes, we need kubeadm, kubectl, kubelet and docker packages upgraded from the upstream repos to 1.16.9.
Packages needed:
It is not entirely required for this round of upgrades, but it is sensible to keep up to date on docker as well:
cri-tools and kubernetes-cni are up to date.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | • Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
| Resolved | • aborrero | T250866 Stage packages for upstream kubeadm v1.16.9 to use in Toolforge |
We have explicit apt pins for these packages, in toolforge::k8s::kubeadmrepo.
So I will try adding the new versions to the repo and hope we don't upgrade by mistake before it is time :-P or that I replace the versions in the repo.
Change 593499 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: kubeadm-k8s: create versioned components
Change 593499 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: kubeadm-k8s: create versioned components
Mentioned in SAL (#wikimedia-operations) [2020-05-04T10:30:32Z] <arturo> running aborrero@apt1001:~ $ sudo -i reprepro --delete clearvanished to cleanup buster-wikimedia|thirdparty/kubeadm-k8s (T250866)
Change 594129 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: kubeadm-k8s: fix docker.io component
Change 594129 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: kubeadm-k8s: fix docker.io component
Mentioned in SAL (#wikimedia-operations) [2020-05-04T10:49:25Z] <arturo> update packages in buster-wikimedia | thirdparty/kubead-k8s-1-15 and thirdparty/kubeadm-k8s-1-16 (T250866)
Mentioned in SAL (#wikimedia-cloud) [2020-05-04T16:43:04Z] <arturo> removing the now unused /etc/apt/sources.list.d/toolforge-k8s-kubeadmrepo.list file (T250866)
Mentioned in SAL (#wikimedia-cloud) [2020-05-04T16:46:03Z] <arturo> removing the now unused /etc/apt/preferences.d/toolforge_k8s_kubeadmrepo* files (T250866)
Change 594925 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] kubeadm: remove package_from_component define
Change 594926 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] wmcs: kubeadm: introduce hiera support for selecting repo component
Mentioned in SAL (#wikimedia-cloud) [2020-05-07T11:12:33Z] <arturo> livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 (T251297 and T250866)
Change 594945 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] wmcs: kubeadm: introduce support for selecting repository component
Mentioned in SAL (#wikimedia-cloud) [2020-05-07T16:33:00Z] <arturo> livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 (T251297 and T250866)
Change 594945 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] wmcs: kubeadm: introduce support for selecting repository component
Change 594926 abandoned by Arturo Borrero Gonzalez:
wmcs: kubeadm: introduce hiera support for selecting repo component
Reason:
merging https://gerrit.wikimedia.org/r/c/operations/puppet/ /594945 instead
Change 594925 abandoned by Arturo Borrero Gonzalez:
kubeadm: remove package_from_component define
Reason:
merging https://gerrit.wikimedia.org/r/c/operations/puppet/ /594945 instead
This is now done.
The trick is to use this hiera key:
profile::wmcs::kubeadm::component: thirdparty/kubeadm-k8s-1-16
The default is thirdparty/kubeadm-k8s-1-15 if unspecified.
I did a stupid thing in the requirements for this task. Apparently that docker upgrade is not tolerated by kubeadm for version 1.16 and we need to downgrade that per T211096#6158997
I'll make a new task for it and maybe try to sort it out.
This would impact our upgrade plans for Toolforge, so trying to bootstrap over in PAWS is becoming quite useful!
Change 598093 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-kubeadm: kubeadm 1.16 requires docker 18.09
Change 598093 merged by Bstorm:
[operations/puppet@production] toolforge-kubeadm: kubeadm 1.16 requires docker 18.09
Mentioned in SAL (#wikimedia-operations) [2020-05-26T15:08:41Z] <arturo> delete/re-import docker/containerd.io packages in the right version in buster-wikimedia/thirdparty/kubeadm-k8s-1-{15,16} (T250866)