Page MenuHomePhabricator
Create Task
Maniphest T263284

Upgrade Toolforge K8s to 1.17
Closed, ResolvedPublic
Actions

Description

Kubernetes supports the latest three minor versions at any given time. We are currently running 1.16, and with 1.19 now released, that will fall off active development.

It's time to start upgrades to 1.17. There are things to check and validate, at least one python library to upgrade, and things to check before moving forward, of course. This should then be upgraded in PAWS as well.

https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes

Also: https://v1-17.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

Details

SubjectRepoBranchLines +/-
toolforge k8s: upgrade docker and containerdoperations/puppetproduction+3 -23
toolforge-k8s: AdmissionsConfiguration is GA after 1.17operations/puppetproduction+1 -1
cloud: kubeadm: refresh version defaultsoperations/puppetproduction+7 -7
kubeadm: wmcs-k8s-node-upgrade: bump version numbersoperations/puppetproduction+6 -6
aptrepo: add missing update reference for thirdparty/kubeadm-k8s-1-17operations/puppetproduction+1 -0
aptrepo: bootstrap repo for thirdparty/kubedam-k8s-1-17operations/puppetproduction+12 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Bstorm triaged this task as Medium priority.Sep 18 2020, 7:43 PM
• Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 18 2020, 7:43 PM
• Bstorm moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.Sep 18 2020, 7:43 PM
• Bstorm added a parent task: T246122: Upgrade the Toolforge Kubernetes cluster to v1.16.
• Bstorm added a comment.Sep 18 2020, 7:46 PM

Connecting to the previous task for reference. This one probably will not be as complicated because we have fewer deprecated objects.

• Bstorm added a project: Kubernetes.Sep 18 2020, 7:47 PM
• Bstorm added a comment.Sep 30 2020, 4:36 PM

In reviewing the changelog, I've picked up a couple callouts so far:

  • metrics-server gets an upgrade, which could change some bits of the prometheus metrics
  • kube-proxy now supports nftables AND autodetects which mode to use
  • ipv6 dualstack support is improved
  • We can upgrade docker
  • kubeadm.k8s.io/v1beta1 is removed (which is fine because we use v1beta2!)
  • default service IP CIDR and must be defined -- I think we already do, but worth checking

That makes it all seem relatively painless. Deploying in toolsbeta will highlight things missed. I'll check if there are useful updates (or important compat ones) to the ingress controllers or calico.

• Bstorm added a comment.Sep 30 2020, 5:21 PM

If we bring calico up to date we will:

  • Be sad that calico doesn't make past-version release notes available for some reason (intentional or not)
  • Be able to try out encrypting all pod traffic if we want, which would vastly increase secrets security in Toolforge if it is performant (and if we keep putting off ipv6, which isn't supported with this) https://docs.projectcalico.org/security/encrypt-cluster-pod-traffic
  • See some bug fixes that might improve performance. I cannot find anything breaking.
• Bstorm added a comment.EditedSep 30 2020, 5:45 PM

On the ingress:

  • We appear to be behind a bit more than we are in calico. Current release is 0.35.0 (we are at 0.25.1)
  • Development is clearly focused on the helm chart, which would be a very good way to deploy and manage the controller (allowing more easy testing locally and things like that). It might be worth proposing that on another task. A *really* old version is even in the internal chart museum. The upstream repo is quietly structured as a helm chart (as of some upgrade since ours). Good info here, including zero downtime upgrade guides: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx Releases are based on the version of the helm chart, which is why the version seems weird vs. their release tags--though they also tag the controller version.
  • TLSv1.3 is enabled by default in 0.33.0
  • We MUST upgrade to 0.32.0 before Kubernetes 1.18 (bugs)
  • Some of the best fixes are in 0.26.0
  • Full update is quite a jump in nginx versions. It'd take some testing, but I think our usage is not too "strange" or customized vs standard capabilities.
  • 0.31.0 and 0.28.0 fix CVEs and fairly serious ones

I'll make a separate task for upgrading the ingress and investigating deploy via helm values file instead of what we do with kubectl.

gerritbot added a comment.Oct 1 2020, 9:52 AM

Change 631410 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: bootstrap repo for thirdparty/kubedam-k8s-1-17

https://gerrit.wikimedia.org/r/631410

gerritbot added a project: Patch-For-Review.Oct 1 2020, 9:52 AM
gerritbot added a comment.Oct 1 2020, 11:06 AM

Change 631410 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: bootstrap repo for thirdparty/kubedam-k8s-1-17

https://gerrit.wikimedia.org/r/631410

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2020, 11:10 AM
Stashbot added a comment.Oct 1 2020, 11:14 AM

Mentioned in SAL (#wikimedia-operations) [2020-10-01T11:14:26Z] <arturo> pulling packages into reprepro for buster-wikimedia/thirdpardy/kubeadm-k8s-1-17 (T263284)

gerritbot added a comment.Oct 1 2020, 11:18 AM

Change 631424 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: add missing update reference for thirdparty/kubeadm-k8s-1-17

https://gerrit.wikimedia.org/r/631424

gerritbot added a project: Patch-For-Review.Oct 1 2020, 11:18 AM

Change 631424 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: add missing update reference for thirdparty/kubeadm-k8s-1-17

https://gerrit.wikimedia.org/r/631424

aborrero subscribed.Oct 1 2020, 11:24 AM

The 1.17 packages are in the repo.

NOTE: the profile::wmcs::kubeadm::component hiera key in the affected nodes should be set accordingly to thirdparty/kubeadm-k8s-1-17 before attempting the upgrade.
Maintenance_bot removed a project: Patch-For-Review.Oct 1 2020, 12:11 PM
• Bstorm mentioned this in T264221: Upgrade the nginx ingress controller in Toolforge (and likely PAWS).Oct 1 2020, 2:50 PM
• Bstorm added a comment.Oct 23 2020, 12:06 AM

I've verified that maintain-kubeusers is able to pass all its tests (which includes all API interactions, I think) against 1.17.13.

That's my biggest concern before deploying straight to toolsbeta!

• Bstorm updated the task description. (Show Details)Oct 29 2020, 10:13 PM
Stashbot added a comment.Oct 29 2020, 10:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-29T22:21:29Z] <bstorm> updated packages for thirdparty/kubeadm-k8s-1-17 to prepare for install T263284

• Bstorm added a comment.Oct 29 2020, 10:27 PM

Ok, now we have 1.17.13 in our repo.

• Bstorm updated the task description. (Show Details)Nov 6 2020, 7:33 PM
Stashbot added a comment.Nov 6 2020, 9:23 PM

Mentioned in SAL (#wikimedia-cloud) [2020-11-06T21:23:01Z] <bstorm> upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 T263284

• Bstorm added a comment.Nov 6 2020, 10:17 PM

On the first node in toolsbeta, I had to manually create a /etc/kubernetes/bootstrap-kubelet.conf when I'd mixed up my steps a little. It was very annoying, but it was easy to copy from another node. Just a note in case we see it again. We should make sure we do not update kubeadm before running kubeadm upgrade plan v1.17.13 apparently. :)

• Bstorm added a comment.Nov 6 2020, 10:17 PM

Another note: there were a lot of mixed up files in toolsbeta's cluster's /etc/apt/sources.list.d that I cleaned up. Puppet did not put them back at least.

gerritbot added a comment.Nov 6 2020, 10:58 PM

Change 639881 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge k8s: upgrade docker and containerd

https://gerrit.wikimedia.org/r/639881

gerritbot added a project: Patch-For-Review.Nov 6 2020, 10:58 PM
gerritbot added a comment.Nov 6 2020, 11:02 PM

Change 639883 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: AdmissionsConfiguration is GA after 1.17

https://gerrit.wikimedia.org/r/639883

Stashbot added a comment.Nov 6 2020, 11:44 PM

Mentioned in SAL (#wikimedia-cloud) [2020-11-06T23:44:48Z] <bstorm> toolsbeta k8s cluster fully upgraded to 1.17.13 T263284

gerritbot added a comment.Nov 10 2020, 10:04 AM

Change 640356 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] kubeadm: wmcs-k8s-node-upgrade: bump version numbers

https://gerrit.wikimedia.org/r/640356

gerritbot added a comment.Nov 10 2020, 10:05 AM

Change 640356 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] kubeadm: wmcs-k8s-node-upgrade: bump version numbers

https://gerrit.wikimedia.org/r/640356

gerritbot added a comment.Nov 30 2020, 12:21 PM

Change 644202 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloud: kubeadm: refresh version defaults

https://gerrit.wikimedia.org/r/644202

gerritbot added a comment.Nov 30 2020, 12:38 PM

Change 644202 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloud: kubeadm: refresh version defaults

https://gerrit.wikimedia.org/r/644202

• Bstorm closed subtask T268669: Upgrade PAWS k8s to 1.17 as Resolved.Nov 30 2020, 6:42 PM
Stashbot added a comment.Dec 10 2020, 11:50 AM

Mentioned in SAL (#wikimedia-cloud) [2020-12-10T11:44:47Z] <arturo> disabled puppet in all k8s nodes in preparation for version upgrade (T263284)

Stashbot added a comment.Dec 10 2020, 11:50 AM

Mentioned in SAL (#wikimedia-cloud) [2020-12-10T11:50:32Z] <arturo> disabled puppet in all k8s nodes in preparation for version upgrade (T263284)

aborrero added a comment.Dec 10 2020, 12:34 PM

Notes are ready here: https://etherpad.wikimedia.org/p/WMCS-2020-12-10-tools-k8s-upgrade

Sent announcement via mailing lists: https://lists.wikimedia.org/pipermail/cloud-announce/2020-December/000348.html

aborrero assigned this task to • Bstorm.Dec 10 2020, 12:34 PM
aborrero moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.
Stashbot added a comment.Dec 10 2020, 12:41 PM

Mentioned in SAL (#wikimedia-cloud) [2020-12-10T12:41:57Z] <arturo> set hiera profile::wmcs::kubeadm::component: thirdparty/kubeadm-k8s-1-17 in project & tools-k8s-control prefix (T263284)

Stashbot added a comment.Dec 10 2020, 3:41 PM

Mentioned in SAL (#wikimedia-cloud) [2020-12-10T15:41:05Z] <arturo> icinga-downtime toolschecker for 2h (T263284)

aborrero closed subtask T269865: k8s: kubelet.conf embeds cert data in k8s < 1.17 as Resolved.Dec 10 2020, 5:01 PM
Stashbot added a comment.Dec 10 2020, 5:16 PM

Mentioned in SAL (#wikimedia-cloud) [2020-12-10T17:16:15Z] <arturo> k8s control nodes were all upgraded to 1.17, now upgrading worker nodes (T263284)

Stashbot added a comment.Dec 10 2020, 5:35 PM

Mentioned in SAL (#wikimedia-cloud) [2020-12-10T17:35:29Z] <bstorm> k8s-control nodes upgraded to 1.17.13 T263284

• Bstorm added a comment.Dec 11 2020, 12:34 AM

The worker nodes are lagging at this point partly due to internet issues on my end.

root@tools-k8s-control-3:~# kubectl get nodes
NAME                  STATUS   ROLES     AGE    VERSION
tools-k8s-control-1   Ready    master    400d   v1.17.13
tools-k8s-control-2   Ready    master    400d   v1.17.13
tools-k8s-control-3   Ready    master    400d   v1.17.13
tools-k8s-ingress-1   Ready    ingress   92d    v1.17.13
tools-k8s-ingress-2   Ready    ingress   92d    v1.17.13
tools-k8s-ingress-3   Ready    ingress   43d    v1.17.13
tools-k8s-worker-30   Ready    <none>    301d   v1.17.13
tools-k8s-worker-31   Ready    <none>    301d   v1.17.13
tools-k8s-worker-32   Ready    <none>    301d   v1.17.13
tools-k8s-worker-33   Ready    <none>    301d   v1.17.13
tools-k8s-worker-34   Ready    <none>    300d   v1.17.13
tools-k8s-worker-35   Ready    <none>    300d   v1.17.13
tools-k8s-worker-36   Ready    <none>    286d   v1.17.13
tools-k8s-worker-37   Ready    <none>    286d   v1.17.13
tools-k8s-worker-38   Ready    <none>    286d   v1.17.13
tools-k8s-worker-39   Ready    <none>    286d   v1.17.13
tools-k8s-worker-40   Ready    <none>    286d   v1.17.13
tools-k8s-worker-41   Ready    <none>    286d   v1.17.13
tools-k8s-worker-42   Ready    <none>    286d   v1.17.13
tools-k8s-worker-43   Ready    <none>    286d   v1.17.13
tools-k8s-worker-44   Ready    <none>    286d   v1.17.13
tools-k8s-worker-45   Ready    <none>    286d   v1.17.13
tools-k8s-worker-46   Ready    <none>    286d   v1.17.13
tools-k8s-worker-47   Ready    <none>    286d   v1.17.13
tools-k8s-worker-48   Ready    <none>    286d   v1.17.13
tools-k8s-worker-49   Ready    <none>    286d   v1.17.13
tools-k8s-worker-50   Ready    <none>    286d   v1.17.13
tools-k8s-worker-51   Ready    <none>    286d   v1.17.13
tools-k8s-worker-52   Ready    <none>    286d   v1.17.13
tools-k8s-worker-53   Ready    <none>    286d   v1.17.13
tools-k8s-worker-54   Ready    <none>    286d   v1.17.13
tools-k8s-worker-55   Ready    <none>    286d   v1.16.10
tools-k8s-worker-56   Ready    <none>    219d   v1.16.10
tools-k8s-worker-57   Ready    <none>    219d   v1.16.10
tools-k8s-worker-58   Ready    <none>    219d   v1.16.10
tools-k8s-worker-59   Ready    <none>    219d   v1.16.10
tools-k8s-worker-60   Ready    <none>    219d   v1.16.10
tools-k8s-worker-61   Ready    <none>    133d   v1.16.10
tools-k8s-worker-62   Ready    <none>    133d   v1.16.10
tools-k8s-worker-64   Ready    <none>    133d   v1.16.10
tools-k8s-worker-65   Ready    <none>    133d   v1.16.10
tools-k8s-worker-66   Ready    <none>    133d   v1.16.10
tools-k8s-worker-67   Ready    <none>    113d   v1.17.13
tools-k8s-worker-68   Ready    <none>    113d   v1.17.13
tools-k8s-worker-69   Ready    <none>    113d   v1.17.13
tools-k8s-worker-70   Ready    <none>    113d   v1.16.10
tools-k8s-worker-71   Ready    <none>    113d   v1.16.10
tools-k8s-worker-72   Ready    <none>    113d   v1.16.10
tools-k8s-worker-73   Ready    <none>    113d   v1.16.10
tools-k8s-worker-74   Ready    <none>    113d   v1.16.10
tools-k8s-worker-75   Ready    <none>    113d   v1.16.10
tools-k8s-worker-76   Ready    <none>    113d   v1.16.10
tools-k8s-worker-77   Ready    <none>    113d   v1.16.10
tools-k8s-worker-78   Ready    <none>    113d   v1.16.10
tools-k8s-worker-79   Ready    <none>    113d   v1.16.10

If it gets too laggy, I'll pause and start it up either later at night or early in the morning when it should finish quite quickly.

Stashbot added a comment.Dec 11 2020, 11:35 AM

Mentioned in SAL (#wikimedia-cloud) [2020-12-11T11:35:13Z] <arturo> uncordon tools-k8s-worker-71 and tools-k8s-worker-55, they weren't uncordoned yesterday for whatever reasons (T263284)

• Bstorm added a comment.Dec 11 2020, 4:44 PM

Upgrade is finally done. However, there were anomalies due to network issues (like uncordoned servers). Going to check a couple things just in case.

• Bstorm closed this task as Resolved.Dec 11 2020, 7:47 PM

Ok, I'm happy that this is done.

bstorm@tools-k8s-control-1:~$ kubectl --as admin --as-group system:masters get nodes -o go-template='{{range .items}}{{.metadata.name}}{{.status.nodeInfo}}{{"\n"}}{{end}}'
tools-k8s-control-1map[architecture:amd64 bootID:af97486f-d320-45d8-a8fa-7317f55a3661 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:9cae425e02d34a48aa4142b6159aeced operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:9cae425e-02d3-4a48-aa41-42b6159aeced]
tools-k8s-control-2map[architecture:amd64 bootID:6aeb8bf2-92b9-47d5-8f84-d14a35182109 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:dc061604c9104462ba68b3be7bd3c95a operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:dc061604-c910-4462-ba68-b3be7bd3c95a]
tools-k8s-control-3map[architecture:amd64 bootID:709136df-468e-474e-88b0-8952995ae76c containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-12-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:62b715ab0cb544da97b5a83c1b07abcf operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:62b715ab-0cb5-44da-97b5-a83c1b07abcf]
tools-k8s-ingress-1map[architecture:amd64 bootID:a47d649d-aaf8-4561-a61f-24b16674eb98 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-12-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:8ebf3c1dd63141eebda93ef162191af9 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:8ebf3c1d-d631-41ee-bda9-3ef162191af9]
tools-k8s-ingress-2map[architecture:amd64 bootID:7ff108da-76de-4c38-9700-0afa25ed5d41 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:3d5ed025a0b246ee85cd9132af6012b2 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:3d5ed025-a0b2-46ee-85cd-9132af6012b2]
tools-k8s-ingress-3map[architecture:amd64 bootID:26c3044e-484d-414b-bd74-6e6a79eaf016 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-12-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:f01145c634ba490c9136a3e44ad2e93e operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:f01145c6-34ba-490c-9136-a3e44ad2e93e]
tools-k8s-worker-30map[architecture:amd64 bootID:76e633b9-6ff0-4386-890e-90def41d7775 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:0afc4e2ab81640828a43bdcbbee3601e operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:0afc4e2a-b816-4082-8a43-bdcbbee3601e]
tools-k8s-worker-31map[architecture:amd64 bootID:e4a3ef6e-bfef-43e2-ac4c-da7b7fbddffb containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:70b83bf4b44d43c5a6a541b31776fd68 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:70b83bf4-b44d-43c5-a6a5-41b31776fd68]
tools-k8s-worker-32map[architecture:amd64 bootID:c60ec1ea-afdc-4cc4-8683-5670804a827b containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:6e3f60eb63754cafa3e4eba0f89f3c44 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:6e3f60eb-6375-4caf-a3e4-eba0f89f3c44]
tools-k8s-worker-33map[architecture:amd64 bootID:04607d98-2c5f-4223-ac79-4aa20f23021f containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:865ee08e5bef4259a82979a39dd75a24 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:865ee08e-5bef-4259-a829-79a39dd75a24]
tools-k8s-worker-34map[architecture:amd64 bootID:afd26a5c-a959-4b2b-80a6-97adc90b0a72 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:d691659c96d44e30bc1d0c3c3f679fdc operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:d691659c-96d4-4e30-bc1d-0c3c3f679fdc]
tools-k8s-worker-35map[architecture:amd64 bootID:c2c28dc5-d7ae-4656-983e-0edcf1faeec7 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:3b44573f0491406f81393d36f7d47ffd operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:3b44573f-0491-406f-8139-3d36f7d47ffd]
tools-k8s-worker-36map[architecture:amd64 bootID:ba0f1d34-a2cd-491b-a769-f508ab383cbc containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:71bd5a4e31fb4d66b91ad4dea4f16147 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:71bd5a4e-31fb-4d66-b91a-d4dea4f16147]
tools-k8s-worker-37map[architecture:amd64 bootID:4061f3de-fc4a-4a2b-92ad-cf819622fb9d containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:f459c79dfa9c4a84aaa064fcdf578e4b operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:f459c79d-fa9c-4a84-aaa0-64fcdf578e4b]
tools-k8s-worker-38map[architecture:amd64 bootID:011e2eb2-467b-4edd-8306-e3f09bfbb911 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:5cceea0a0463427eb718a1bd324285cf operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:5cceea0a-0463-427e-b718-a1bd324285cf]
tools-k8s-worker-39map[architecture:amd64 bootID:53304e6c-a033-46c7-be1a-99f530bf0984 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:780e437cc5d84e06a47dc48a88a0f1bc operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:780e437c-c5d8-4e06-a47d-c48a88a0f1bc]
tools-k8s-worker-40map[architecture:amd64 bootID:f478c63e-2d2a-4a7a-a775-70527ff53868 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:958c1013853543ba81fb71aaebcf60f7 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:958c1013-8535-43ba-81fb-71aaebcf60f7]
tools-k8s-worker-41map[architecture:amd64 bootID:9f01aa60-e01e-4f7c-8189-e96a84fc19bd containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:1c9773864f1947c0b69f46edb8768a50 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:1c977386-4f19-47c0-b69f-46edb8768a50]
tools-k8s-worker-42map[architecture:amd64 bootID:31184e4b-7879-42a3-b93b-fd2ba3a6c392 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:f849b9bce0394cdf8c1021a85e2b69a3 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:f849b9bc-e039-4cdf-8c10-21a85e2b69a3]
tools-k8s-worker-43map[architecture:amd64 bootID:52b73777-3254-4b44-a9f5-193ee21fc2e7 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:cda195d14464416691752b7fd98efeac operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:cda195d1-4464-4166-9175-2b7fd98efeac]
tools-k8s-worker-44map[architecture:amd64 bootID:1d2c2d13-6902-4db1-a42d-aea6e75dc754 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:86b26d82d7c14b6799cf1efb5aaf432d operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:86b26d82-d7c1-4b67-99cf-1efb5aaf432d]
tools-k8s-worker-45map[architecture:amd64 bootID:faecd75c-521c-4907-a41d-024d44ebfce5 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:43dfb4a2e36442889d1edb643df3ac29 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:43dfb4a2-e364-4288-9d1e-db643df3ac29]
tools-k8s-worker-46map[architecture:amd64 bootID:c93f3186-0530-468f-94e7-b3fa5c707e2b containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:b043e03330b84257b1fd48a50b399bad operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:b043e033-30b8-4257-b1fd-48a50b399bad]
tools-k8s-worker-47map[architecture:amd64 bootID:3579a671-c5bf-4050-b2df-031428ec5a98 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:2dcf81fc79a1433f9fccbc7067947a97 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:2dcf81fc-79a1-433f-9fcc-bc7067947a97]
tools-k8s-worker-48map[architecture:amd64 bootID:5777a47a-130a-48c2-adc6-517856ffd2d9 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:e0f3b053d31a402f85fdee98423b3e18 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:e0f3b053-d31a-402f-85fd-ee98423b3e18]
tools-k8s-worker-49map[architecture:amd64 bootID:ed2c0dac-6e9e-48ff-ab18-2426ba5468e2 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:f953c4c57cfa4100823905dabce7c3d5 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:f953c4c5-7cfa-4100-8239-05dabce7c3d5]
tools-k8s-worker-50map[architecture:amd64 bootID:d8c762b0-90bb-4d88-84e7-f47c566c5077 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:9a0fe25cfb9d418793d470e0e74b4832 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:9a0fe25c-fb9d-4187-93d4-70e0e74b4832]
tools-k8s-worker-51map[architecture:amd64 bootID:e517c09d-ba26-4d70-a607-4e29e1bd2b50 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:edbfcb4eb6474888b1a974f1453e6edf operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:edbfcb4e-b647-4888-b1a9-74f1453e6edf]
tools-k8s-worker-52map[architecture:amd64 bootID:5a2d2d3d-ed1a-4303-a157-b616f05b6022 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:4f19382485d843698be3c8b96abbd71d operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:4f193824-85d8-4369-8be3-c8b96abbd71d]
tools-k8s-worker-53map[architecture:amd64 bootID:14398b3b-13a7-4e68-84f5-0fc28761d5c4 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:16c4070fc56e452a98e568a3b15ad2b9 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:16c4070f-c56e-452a-98e5-68a3b15ad2b9]
tools-k8s-worker-54map[architecture:amd64 bootID:e1ce3060-83ef-49db-9652-4f6929b77c43 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:e42596c5f8f74952b1bd5d8ffd04cd5b operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:e42596c5-f8f7-4952-b1bd-5d8ffd04cd5b]
tools-k8s-worker-55map[architecture:amd64 bootID:fa1d1ae1-bf21-45e8-a365-622545a16828 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:4002f61367824d96952ba40462b7a20a operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:4002f613-6782-4d96-952b-a40462b7a20a]
tools-k8s-worker-56map[architecture:amd64 bootID:081e2e27-c3d1-4086-a413-b561f311b46b containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:cb49af9bc1404842b6d18a765bb34de5 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:cb49af9b-c140-4842-b6d1-8a765bb34de5]
tools-k8s-worker-57map[architecture:amd64 bootID:00f12984-0a44-4c51-87f9-6e09e83c7ad5 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:01c81b24d2a643239bc998b7ff73382c operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:01c81b24-d2a6-4323-9bc9-98b7ff73382c]
tools-k8s-worker-58map[architecture:amd64 bootID:d2316abc-70b3-40f8-9268-dab762078e32 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:811e58ef01ca41bca4031f2f81732de9 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:811e58ef-01ca-41bc-a403-1f2f81732de9]
tools-k8s-worker-59map[architecture:amd64 bootID:45622f87-539b-46ec-9ca3-225097a4309a containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:e04194f4992640d6ae75e4e30b92bb2c operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:e04194f4-9926-40d6-ae75-e4e30b92bb2c]
tools-k8s-worker-60map[architecture:amd64 bootID:005ab5ef-cf72-431b-83a4-ac862d2ab6d8 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:928d61d83b3345d0b9d79126d90b3367 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:928d61d8-3b33-45d0-b9d7-9126d90b3367]
tools-k8s-worker-61map[architecture:amd64 bootID:9efb44c7-66de-4c99-9f2d-b2de93d5c193 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:e0d016d375eb4010b5af2e0416494010 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:e0d016d3-75eb-4010-b5af-2e0416494010]
tools-k8s-worker-62map[architecture:amd64 bootID:7e7c6221-17bc-4b46-ad22-077aaed82f50 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:20f38041998d4d2eab0fdcea9022dc43 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:20f38041-998d-4d2e-ab0f-dcea9022dc43]
tools-k8s-worker-64map[architecture:amd64 bootID:5e77d76a-53bc-40d7-b52c-ee198f3a8534 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:792ddffa495c412588a9f9772836bf39 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:792ddffa-495c-4125-88a9-f9772836bf39]
tools-k8s-worker-65map[architecture:amd64 bootID:d48491f0-09a6-4321-96c8-cbb5956b7038 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:c6fa505438a64d0fb4a039202c59ad5f operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:c6fa5054-38a6-4d0f-b4a0-39202c59ad5f]
tools-k8s-worker-66map[architecture:amd64 bootID:a9bda12b-7ba4-402e-acb2-de862965f3cd containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:7a45fd5593d747dc9c0674e665f0bf2b operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:7a45fd55-93d7-47dc-9c06-74e665f0bf2b]
tools-k8s-worker-67map[architecture:amd64 bootID:5ddc18a2-d966-4839-a9ed-714e31b4af87 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:b25d4569b0e24520a61a59bfcafca654 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:b25d4569-b0e2-4520-a61a-59bfcafca654]
tools-k8s-worker-68map[architecture:amd64 bootID:02a455b9-c404-492a-888b-2b7d91033ea4 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:604fde78fba542789a045d687332e161 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:604fde78-fba5-4278-9a04-5d687332e161]
tools-k8s-worker-69map[architecture:amd64 bootID:76a240b7-65a5-41d4-bd13-3dab4c14ab4e containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:6baefc55762e4654aef211dba3e5d954 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:6baefc55-762e-4654-aef2-11dba3e5d954]
tools-k8s-worker-70map[architecture:amd64 bootID:3d78b876-7f81-4743-b44c-c16fef1dc555 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-11-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:444d77386b10475a87043af8cee2a04c operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:444d7738-6b10-475a-8704-3af8cee2a04c]
tools-k8s-worker-71map[architecture:amd64 bootID:a8310b73-2cec-4ec9-9afd-eb36ead7e44d containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:23f6ee3f2b2c447da15ceaeeeee50a42 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:23f6ee3f-2b2c-447d-a15c-eaeeeee50a42]
tools-k8s-worker-72map[architecture:amd64 bootID:4b691251-94bf-4dc6-b75d-4adc3145823d containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:25bc6a5d80d9424fbedc7f626c457e6b operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:25bc6a5d-80d9-424f-bedc-7f626c457e6b]
tools-k8s-worker-73map[architecture:amd64 bootID:cd55ef74-4339-48aa-b509-f823ac93c2a6 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:997139f442ae425091a9688fba413f7b operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:997139f4-42ae-4250-91a9-688fba413f7b]
tools-k8s-worker-74map[architecture:amd64 bootID:1a0c3dbc-0278-4868-ba63-a97154dde1ee containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:98a14ff9519e4873bb29b17abe072374 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:98a14ff9-519e-4873-bb29-b17abe072374]
tools-k8s-worker-75map[architecture:amd64 bootID:bd48d7c2-1d3a-4aea-8ca9-6f0955cfec0a containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:47c1fdba747545c5a656801ce2bf11cd operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:47c1fdba-7475-45c5-a656-801ce2bf11cd]
tools-k8s-worker-76map[architecture:amd64 bootID:cdf1c5a2-a023-407d-9c8a-f211e7c334dd containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:4e49205125a34442b8b91959f42917fe operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:4e492051-25a3-4442-b8b9-1959f42917fe]
tools-k8s-worker-77map[architecture:amd64 bootID:8ec29f11-cba8-434c-a361-0f6dce8f741e containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:a334a5589e564eb08525ca261f3366c9 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:a334a558-9e56-4eb0-8525-ca261f3366c9]
tools-k8s-worker-78map[architecture:amd64 bootID:cd761eb8-9b64-4ca5-bb40-cec18079f899 containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:9ed684fe2c7f42e98d8e4c4229673662 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:9ed684fe-2c7f-42e9-8d8e-4c4229673662]
tools-k8s-worker-79map[architecture:amd64 bootID:e47e8719-d404-4ae7-ad67-85d4a91455ac containerRuntimeVersion:docker://18.9.9 kernelVersion:4.19.0-10-amd64 kubeProxyVersion:v1.17.13 kubeletVersion:v1.17.13 machineID:010a4125318b467fb951bddd74cf89c1 operatingSystem:linux osImage:Debian GNU/Linux 10 (buster) systemUUID:010a4125-318b-467f-b951-bddd74cf89c1]
gerritbot added a comment.Jan 22 2021, 12:06 AM

Change 639883 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: AdmissionsConfiguration is GA after 1.17

https://gerrit.wikimedia.org/r/639883

gerritbot added a comment.Jan 26 2021, 1:37 PM

Change 639881 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge k8s: upgrade docker and containerd

https://gerrit.wikimedia.org/r/639881

Stashbot added a comment.Jan 26 2021, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-26T13:41:16Z] <arturo> admin update some kubernetes-related packages in buster-wikimedia/thirdparty/kubeadm-k8s-1-17 (T263284)

aborrero added a comment.Jan 26 2021, 1:43 PM

@Bstorm now we can upgrade packages in each node when you see fit:

aborrero@apt1001:~ $ sudo -i reprepro --component thirdparty/kubeadm-k8s-1-17 checkupdate buster-wikimedia
Calculating packages to get...
Updates needed for 'buster-wikimedia|thirdparty/kubeadm-k8s-1-17|amd64':
'containerd.io': '1.2.13-2' will be upgraded to '1.4.3-1' (from 'thirdparty/kubeadm-k8s-docker.com'):
 files needed: pool/thirdparty/kubeadm-k8s-1-17/c/containerd.io/containerd.io_1.4.3-1_amd64.deb
'docker-ce': '5:18.09.9~3-0~debian-stretch' will be upgraded to '5:19.03.14~3-0~debian-stretch' (from 'thirdparty/kubeadm-k8s-docker.com'):
 files needed: pool/thirdparty/kubeadm-k8s-1-17/d/docker-ce/docker-ce_19.03.14~3-0~debian-stretch_amd64.deb
'docker-ce-cli': '5:18.09.9~3-0~debian-stretch' will be upgraded to '5:19.03.14~3-0~debian-stretch' (from 'thirdparty/kubeadm-k8s-docker.com'):
 files needed: pool/thirdparty/kubeadm-k8s-1-17/d/docker-ce/docker-ce-cli_19.03.14~3-0~debian-stretch_amd64.deb
'helm': '3.4.0-1' will be upgraded to '3.5.0-1' (from 'thirdparty/helm3'):
 files needed: pool/thirdparty/kubeadm-k8s-1-17/h/helm/helm_3.5.0-1_amd64.deb
'kubeadm': '1.17.13-00' will be upgraded to '1.17.17-00' (from 'thirdparty/kubeadm-k8s-kubernetes.io-1-17'):
 files needed: pool/thirdparty/kubeadm-k8s-1-17/k/kubeadm/kubeadm_1.17.17-00_amd64.deb
'kubectl': '1.17.13-00' will be upgraded to '1.17.17-00' (from 'thirdparty/kubeadm-k8s-kubernetes.io-1-17'):
 files needed: pool/thirdparty/kubeadm-k8s-1-17/k/kubectl/kubectl_1.17.17-00_amd64.deb
'kubelet': '1.17.13-00' will be upgraded to '1.17.17-00' (from 'thirdparty/kubeadm-k8s-kubernetes.io-1-17'):
 files needed: pool/thirdparty/kubeadm-k8s-1-17/k/kubelet/kubelet_1.17.17-00_amd64.deb
Maintenance_bot removed a project: Patch-For-Review.Jan 26 2021, 2:10 PM
taavi closed subtask T264221: Upgrade the nginx ingress controller in Toolforge (and likely PAWS) as Resolved.Jul 14 2021, 10:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits