Page MenuHomePhabricator
Create Task
Maniphest T280299

Upgrade Toolforge Kubernetes to latest 1.18
Closed, ResolvedPublic
Actions

Description

Always start with toolsbeta.

The current release of Kubernetes is 1.21. We aim to stay within the supported versions, and while there is currently a 1 year support cycle starting at 1.20, we are at 1.17. On those versions only three minor versions were supported for patches, so we've dropped clean off. We should try to move to 1.19 as soon as is feasible as well.

Make sure you read https://v1-18.docs.kubernetes.io/docs/setup/release/notes/#urgent-upgrade-notes
Then there's the appropriate upgrade guide: https://v1-18.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

And then there's our notes: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes

Details

SubjectRepoBranchLines +/-
tools-clush: remove paws from clush and add the rest of the k8s setupoperations/puppetproduction+3 -2
aptrepo: bootstrap repo for thirdparty/kubedam-k8s-1-18operations/puppetproduction+14 -3
Update to apps/v1labs/tools/registry-admission-webhookmaster+5 -2
golang fun: fix the module entries for the k8s api codelabs/tools/registry-admission-webhookmaster+102 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Bstorm triaged this task as Medium priority.Apr 15 2021, 10:13 PM
• Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 15 2021, 10:13 PM
• Bstorm updated the task description. (Show Details)Apr 15 2021, 10:31 PM
• Bstorm added subscribers: taavi, Andrew.Apr 15 2021, 10:35 PM

If we get to 1.19, it's the exact same process, but there is no need to refresh the certs again if done in the same six month period.

taavi added a comment.Apr 16 2021, 8:12 AM

I've read the release notes for both 1.18 and 1.19, summary below.

Kubernetes

1.18
  • consumers of the 'certificatesigningrequests/approval' API must now have permission to 'approve' CSRs for the specific signer requested by the CSR. More information on the new signerName field and the required authorization can be found at https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests#authorization (#88246, @munnerz) [SIG API Machinery, Apps, Auth, CLI, Node and Testing]
    • This breaks maintain-kubeusers, see patch attached on T280300
  • Deprecated user-facing APIs removed, all have replacements on newer APIs, just need to check if any tools are using the legacy ones
    • apps/v1beta[12]
    • extensions/v1beta1 daemonsets, deployments, replicasets
    • extensions/v1beta1 networkpolicies
    • extensions/v1beta1 podsecuritypolicies
  • Upgrade guide itself looks fairly standard, nothing special in there
1.19
  • I don't see anything special on the release notes or upgrade guide, should be a relatively simple upgrade except that we need to upgrade Calico before upgrading to Kubernetes 1.19 (see below).

Calico

  • Docs for older versions not available online? ended up digging git tags for their source on github
  • We're currently on Calico 3.14, which supports k8s 1.16-1.18, latest Calico (3.18) supports k8s 1.18-1.20
  • Do we need to upgrade via each version or can we skip some from the middle?
gerritbot added a comment.Apr 16 2021, 8:22 AM

Change 680253 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] aptrepo: bootstrap repo for thirdparty/kubedam-k8s-1-17

https://gerrit.wikimedia.org/r/680253

gerritbot added a project: Patch-For-Review.Apr 16 2021, 8:22 AM
taavi added a parent task: T280340: Upgrade Toolforge Kubernetes to latest 1.19.Apr 16 2021, 10:48 AM
taavi added a comment.Apr 16 2021, 1:39 PM

One more thing from 1.18 notes:

  • New IngressClass resource and a field to replace now-deprecated field kubernetes.io/ingress.class, field is not in use in webservice but is used in the iw tool and in its documentation and possibly on other hand-crafted deployments
gerritbot added a comment.Apr 16 2021, 3:02 PM

Change 680367 had a related patch set uploaded (by Majavah; author: Majavah):

[labs/tools/registry-admission-webhook@master] Update to apps/v1

https://gerrit.wikimedia.org/r/680367

• Bstorm added a comment.Apr 16 2021, 3:46 PM
In T280299#7008150, @Majavah wrote:

I've read the release notes for both 1.18 and 1.19, summary below.

Kubernetes

1.18
  • Deprecated user-facing APIs removed, all have replacements on newer APIs, just need to check if any tools are using the legacy ones
    • apps/v1beta[12]
    • extensions/v1beta1 daemonsets, deployments, replicasets
    • extensions/v1beta1 networkpolicies
    • extensions/v1beta1 podsecuritypolicies
  • Upgrade guide itself looks fairly standard, nothing special in there

This bit is likely ok except for new objects. The apis stop being served, but the server translates existing objects to the current APIs (usually). The rare cases that isn't true usually involve removing an entire feature.

• Bstorm added a comment.Apr 16 2021, 3:46 PM

Anyone doing this upgrade may see etcd timeout failures due to high iowait on the etcd cluster, btw. T279723 might help, but this has been an on-going problem.

gerritbot added a comment.Apr 16 2021, 5:09 PM

Change 680388 had a related patch set uploaded (by Bstorm; author: Bstorm):

[labs/tools/registry-admission-webhook@master] golang fun: fix the module entries for the k8s api code

https://gerrit.wikimedia.org/r/680388

gerritbot added a comment.Apr 16 2021, 5:15 PM

Change 680388 merged by jenkins-bot:

[labs/tools/registry-admission-webhook@master] golang fun: fix the module entries for the k8s api code

https://gerrit.wikimedia.org/r/680388

• Bstorm mentioned this in rLTRA14f913979ca6: golang fun: fix the module entries for the k8s api code.Apr 16 2021, 5:17 PM
gerritbot added a comment.Apr 16 2021, 5:33 PM

Change 680367 merged by jenkins-bot:

[labs/tools/registry-admission-webhook@master] Update to apps/v1

https://gerrit.wikimedia.org/r/680367

taavi mentioned this in rLTRA7d9cc0d1a012: Update to apps/v1.Apr 16 2021, 5:36 PM
taavi added a subtask: T264221: Upgrade the nginx ingress controller in Toolforge (and likely PAWS).Apr 17 2021, 6:32 AM
gerritbot added a comment.Apr 19 2021, 10:00 AM

Change 680253 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] aptrepo: bootstrap repo for thirdparty/kubedam-k8s-1-18

https://gerrit.wikimedia.org/r/680253

Maintenance_bot removed a project: Patch-For-Review.Apr 19 2021, 10:10 AM
Stashbot added a comment.Jun 8 2021, 2:08 PM

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T14:08:37Z] <majavah> update toolsbeta-test-k8s-control-4 to kubernetes 1.18 T280299

Stashbot added a comment.Jun 8 2021, 2:57 PM

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T14:57:30Z] <majavah> continuing to update rest of k8s control nodes T280299

Stashbot added a comment.Jun 8 2021, 3:02 PM

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T15:02:20Z] <majavah> continuing to update k8s ingress nodes T280299

Stashbot added a comment.Jun 8 2021, 3:11 PM

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T15:11:56Z] <majavah> updating k8s worker nodes to 1.18 T280299

taavi closed subtask T280301: Refresh certs that are not controlled by kubeadm as Resolved.Jun 8 2021, 6:39 PM

I upgraded toolsbeta today and tested some basic operations. Everything seems to be working fine, I don't see an issue continuing to tools/paws as long as new tool creation works which I don't have access to test.

taavi closed subtask T280300: Validate that maintain-kubeusers works with k8s 1.18 as Resolved.Jun 9 2021, 5:39 AM
taavi claimed this task.Jun 9 2021, 6:54 PM
Restricted Application added a project: User-Majavah. · View Herald TranscriptJun 9 2021, 6:54 PM
taavi moved this task from Unsorted to Working on on the User-Majavah board.Jun 20 2021, 11:29 AM
taavi added a comment.Jun 23 2021, 5:08 AM

Upgrade scheduled for 2021-06-29: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/BTDUMJML2K7AMGAUUKAGGCWCKLWW6HI4/

gerritbot added a comment.Jun 28 2021, 8:38 PM

Change 701975 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] tools-clush: remove paws from clush and add the rest of the k8s setup

https://gerrit.wikimedia.org/r/701975

gerritbot added a project: Patch-For-Review.Jun 28 2021, 8:38 PM
gerritbot added a comment.Jun 29 2021, 3:05 PM

Change 701975 merged by Bstorm:

[operations/puppet@production] tools-clush: remove paws from clush and add the rest of the k8s setup

https://gerrit.wikimedia.org/r/701975

Maintenance_bot removed a project: Patch-For-Review.Jun 29 2021, 3:10 PM
Stashbot added a comment.Jun 29 2021, 5:03 PM

Mentioned in SAL (#wikimedia-cloud) [2021-06-29T17:03:02Z] <majavah> starting toolforge kubernetes 1.18 upgrade - T280299

taavi closed this task as Resolved.Jun 29 2021, 8:10 PM
Stashbot added a comment.Jun 29 2021, 8:11 PM

Mentioned in SAL (#wikimedia-cloud) [2021-06-29T20:11:49Z] <majavah> toolforge kubernetes upgrade complete T280299

taavi closed subtask T264221: Upgrade the nginx ingress controller in Toolforge (and likely PAWS) as Resolved.Jul 14 2021, 10:43 AM
taavi closed subtask T280302: Upgrade PAWS Kubernetes to the latest 1.18 release as Resolved.Jul 21 2021, 4:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL