Page MenuHomePhabricator
Create Task
Maniphest T246689

Toolforge: cleanup legacy kubernetes cluster
Closed, ResolvedPublic
Actions

Description

Some stuff to clean, in the legacy kubernetes cluster specifically:

  • make sure T246519: "paws-public" tool running 2 custom pods on legacy Kubernetes cluster is solved.
  • tools-proxy nodes usage of the cluster
  • bunch of empty worker nodes.
  • etcd servers (and make sure nothing in paws is using it!)
  • flannel stuff, no longer in use by kubernetes.
  • the entire toollabs module (once the 'master' and workers are down) - they should be the last things actually using it
  • monitoring / prometheus metrics
  • grafana dashboards that only work with the old cluster (they won't get anything from prometheus anymore)
  • any other mechanism interacting with the legacy k8s cluster
  • get rid of old non-sssd docker containers in the registry

Details

SubjectRepoBranchLines +/-
toolforge cleanup: remove the ferm_handlers profileoperations/puppetproduction+0 -44
toolforge: clean up the now-unneeded ferm_handlersoperations/puppetproduction+4 -8
k8s: purge flannel from the environmentoperations/puppetproduction+0 -80
Remove temporary code from 2020 Kubernetes migrationoperations/software/tools-webservicemaster+73 -176
toolforge: remove the last toollabs roleoperations/puppetproduction+0 -12
toolserver: refactor into profile and move off "toollabs" nameoperations/puppetproduction+15 -2
toolforge: refactor docker builder to remove toollabs moduleoperations/puppetproduction+8 -13
toolforge: fix file location for grid override.my.cnfoperations/puppetproduction+3 -1
toolforge: remove almost entire toollabs module and related rolesoperations/puppetproduction+0 -3 K
toolforge-k8s: remove old legacy-cluster code from bastionoperations/puppetproduction+0 -6
toolforge-clush: correct the classifications and remove legacy k8soperations/puppetproduction+11 -14
toolforge: remove old k8s client material for Jessieoperations/puppetproduction+0 -23
toolschecker: Update to monitor the new etcd clusteroperations/puppetproduction+49 -5
tools-prometheus: removing material related to the legacy k8s clusteroperations/puppetproduction+0 -76
toolforge: remove monitoring for old k8s cluster nodes and flannel etcdoperations/puppetproduction+1 -38
toolforge: clean up maintain_kubeusers and legacy proxy puppet code.operations/puppetproduction+0 -1 K
toolforge: remove special configuration for kubernetes on proxy serversoperations/puppetproduction+15 -29
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
bd808 triaged this task as High priority.Mar 2 2020, 5:39 PM
Stashbot added a comment.Mar 3 2020, 10:51 AM

Mentioned in SAL (#wikimedia-cloud) [2020-03-03T10:51:04Z] <arturo> cordoned/drained all legacy k8s worker nodes except 1001/1002 (T246689)

Stashbot added a comment.Mar 3 2020, 10:54 AM

Mentioned in SAL (#wikimedia-cloud) [2020-03-03T10:54:53Z] <arturo> deleted VMs tools-worker-[1003-1020] (legacy k8s cluster) (T246689)

gerritbot added a comment.Mar 4 2020, 10:44 AM

Change 576469 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Bstorm):
[operations/puppet@production] toolforge: remove special configuration for kubernetes on proxy servers

https://gerrit.wikimedia.org/r/576469

gerritbot added a project: Patch-For-Review.Mar 4 2020, 10:44 AM
bd808 updated the task description. (Show Details)Mar 4 2020, 4:17 PM
aborrero updated the task description. (Show Details)Mar 4 2020, 4:26 PM
gerritbot added a comment.Mar 4 2020, 5:26 PM

Change 576469 merged by Bstorm:
[operations/puppet@production] toolforge: remove special configuration for kubernetes on proxy servers

https://gerrit.wikimedia.org/r/576469

gerritbot added a comment.Mar 4 2020, 5:44 PM

Change 576911 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: clean up maintain_kubeusers and legacy proxy puppet code.

https://gerrit.wikimedia.org/r/576911

gerritbot added a comment.Mar 4 2020, 6:14 PM

Change 576911 merged by Bstorm:
[operations/puppet@production] toolforge: clean up maintain_kubeusers and legacy proxy puppet code.

https://gerrit.wikimedia.org/r/576911

• Bstorm updated the task description. (Show Details)Mar 4 2020, 6:32 PM

Remove gridengine as a possible vector for flannel because it was only on the proxies and k8s. It is now removed from the proxies.

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2020, 7:10 PM
gerritbot added a comment.Mar 4 2020, 11:16 PM

Change 576992 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: remove monitoring for old k8s cluster nodes and flannel etcd

https://gerrit.wikimedia.org/r/576992

gerritbot added a project: Patch-For-Review.Mar 4 2020, 11:16 PM
gerritbot added a comment.Mar 4 2020, 11:36 PM

Change 576995 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: remove old k8s client material for Jessie

https://gerrit.wikimedia.org/r/576995

gerritbot added a comment.Mar 5 2020, 3:18 PM

Change 576992 merged by Bstorm:
[operations/puppet@production] toolforge: remove monitoring for old k8s cluster nodes and flannel etcd

https://gerrit.wikimedia.org/r/576992

gerritbot added a comment.Mar 5 2020, 3:41 PM

Change 577261 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] tools-prometheus: removing material related to the legacy k8s cluster

https://gerrit.wikimedia.org/r/577261

• Bstorm updated the task description. (Show Details)Mar 5 2020, 4:02 PM
• Bstorm added a comment.Mar 5 2020, 4:11 PM

paws etcd is entirely in a container on the "master" node:

root      1158  1118 17 Jan09 ?        9-17:09:11 kube-apiserver --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --advertise-address=172.16.2.205 --service-account-key-file=/etc/kubernetes/pki/sa.pub --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --insecure-port=0 --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-allowed-names=front-proxy-client --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota --requestheader-username-headers=X-Remote-User --service-cluster-ip-range=10.96.0.0/12 --client-ca-file=/etc/kubernetes/pki/ca.crt --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --secure-port=6443 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --enable-bootstrap-token-auth=true --allow-privileged=true --authorization-mode=Node,RBAC --etcd-servers=http://127.0.0.1:2379

So no worries there.

gerritbot added a comment.Mar 5 2020, 4:30 PM

Change 577279 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-clush: correct the classifications and remove legacy k8s

https://gerrit.wikimedia.org/r/577279

gerritbot added a comment.Mar 5 2020, 6:40 PM

Change 577261 merged by Bstorm:
[operations/puppet@production] tools-prometheus: removing material related to the legacy k8s cluster

https://gerrit.wikimedia.org/r/577261

• Bstorm added a comment.Mar 5 2020, 7:07 PM

That removed prometheus metrics.

• Bstorm updated the task description. (Show Details)Mar 5 2020, 7:08 PM
gerritbot added a comment.Mar 5 2020, 9:09 PM

Change 577341 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolschecker: Update to monitor the new etcd cluster

https://gerrit.wikimedia.org/r/577341

gerritbot added a comment.Mar 5 2020, 10:23 PM

Change 577341 merged by Bstorm:
[operations/puppet@production] toolschecker: Update to monitor the new etcd cluster

https://gerrit.wikimedia.org/r/577341

• Bstorm updated the task description. (Show Details)Mar 5 2020, 10:30 PM
gerritbot added a comment.Mar 6 2020, 8:40 PM

Change 576995 merged by Bstorm:
[operations/puppet@production] toolforge: remove old k8s client material for Jessie

https://gerrit.wikimedia.org/r/576995

gerritbot added a comment.Mar 10 2020, 5:26 AM

Change 578408 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Remove temporary code from 2020 Kubernetes migration

https://gerrit.wikimedia.org/r/578408

Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Mar 10 2020, 4:18 PM
gerritbot added a comment.Mar 10 2020, 6:40 PM

Change 577279 merged by Bstorm:
[operations/puppet@production] toolforge-clush: correct the classifications and remove legacy k8s

https://gerrit.wikimedia.org/r/577279

bd808 moved this task from Backlog to WMCS infrastructure on the Cloud-VPS (Debian Jessie Deprecation) board.Mar 12 2020, 12:45 AM
Stashbot added a comment.Mar 16 2020, 7:06 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-16T19:06:10Z] <bstorm_> shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd T246689

Stashbot added a comment.Mar 16 2020, 7:07 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-16T19:07:50Z] <bstorm_> shutting down toolsbeta-flannel-etcd-01 T246689

Stashbot added a comment.Mar 16 2020, 7:45 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-16T19:45:01Z] <bstorm_> deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and toolsbeta-k8s-etcd-01 T246689

Stashbot added a comment.Mar 16 2020, 9:38 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-16T21:38:28Z] <bstorm_> removed lots of hiera related to the legacy k8s cluster T246689

Stashbot added a comment.Mar 16 2020, 9:59 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-16T21:59:42Z] <bstorm_> shut down tools-worker-1001 and tools-worker-1002 T246689

Stashbot added a comment.Mar 16 2020, 10:00 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-16T22:00:23Z] <bstorm_> shut off tools-k8s-master-01 T246689

Stashbot added a comment.Mar 16 2020, 10:01 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-16T22:01:24Z] <bstorm_> shut off tools-k8s-etcd-01/02/03 T246689

gerritbot added a comment.Mar 16 2020, 10:13 PM

Change 580134 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: remove old legacy-cluster code from bastion

https://gerrit.wikimedia.org/r/580134

gerritbot added a comment.Mar 16 2020, 11:03 PM

Change 580134 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: remove old legacy-cluster code from bastion

https://gerrit.wikimedia.org/r/580134

Stashbot added a comment.Mar 17 2020, 12:08 AM

Mentioned in SAL (#wikimedia-cloud) [2020-03-17T00:08:05Z] <bstorm_> shut off tools-flannel-etcd-01/02/03 T246689

• Bstorm added a comment.Mar 17 2020, 12:11 AM

Ok, the entire legacy cluster in tools, including flannel is shut down. I'll leave that overnight and delete things more in the morning. In toolsbeta, things are deleted and hiera values removed.

• Bstorm added a comment.Mar 17 2020, 12:11 AM

PAWS is still up and working, so that's good :)

Stashbot added a comment.Mar 18 2020, 4:59 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-18T16:59:40Z] <bstorm_> deleting "tools-worker-1002", "tools-worker-1001", "tools-k8s-master-01", "tools-flannel-etcd-03", "tools-k8s-etcd-03", "tools-flannel-etcd-02", "tools-k8s-etcd-02", "tools-flannel-etcd-01", "tools-k8s-etcd-01" T246689

• Bstorm updated the task description. (Show Details)Mar 18 2020, 5:07 PM

By "flannel stuff", I'm going to assume we mean the puppet code.

Stashbot added a comment.Mar 18 2020, 5:36 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-18T17:36:01Z] <bstorm_> removed lots of deprecated hiera keys from horizon for the old cluster T246689

• Bstorm added a comment.Mar 18 2020, 5:52 PM

Noticed that the ToolsDB cluster wasn't sending prometheus metrics because the security group needed an update when testing puppet after stripping out a lot of old hiera. Fixed that.

Stashbot added a comment.Mar 18 2020, 5:57 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-18T17:57:39Z] <bstorm_> removed puppet prefix tools-k8s-master T246689

Stashbot added a comment.Mar 18 2020, 5:58 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-18T17:58:07Z] <bstorm_> removed puppet prefix tools-worker T246689

Stashbot added a comment.Mar 18 2020, 6:04 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-18T18:04:37Z] <bstorm_> removed puppet prefix tools-flannel-etcd T246689

gerritbot added a comment.Mar 18 2020, 7:15 PM

Change 581056 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: remove the entire toollabs module and all related roles

https://gerrit.wikimedia.org/r/581056

gerritbot added a comment.Mar 19 2020, 4:36 PM

Change 581056 merged by Bstorm:
[operations/puppet@production] toolforge: remove almost entire toollabs module and related roles

https://gerrit.wikimedia.org/r/581056

gerritbot added a comment.Mar 19 2020, 4:45 PM

Change 581647 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: refactor docker builder to remove toollabs module

https://gerrit.wikimedia.org/r/581647

gerritbot added a comment.Mar 19 2020, 5:01 PM

Change 581654 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolserver: refactor into profile and move off "toollabs" name

https://gerrit.wikimedia.org/r/581654

gerritbot added a comment.Mar 19 2020, 5:50 PM

Change 581673 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: fix file location for grid override.my.cnf

https://gerrit.wikimedia.org/r/581673

gerritbot added a comment.Mar 19 2020, 5:55 PM

Change 581673 merged by Bstorm:
[operations/puppet@production] toolforge: fix file location for grid override.my.cnf

https://gerrit.wikimedia.org/r/581673

gerritbot added a comment.Mar 19 2020, 6:47 PM

Change 581647 merged by Bstorm:
[operations/puppet@production] toolforge: refactor docker builder to remove toollabs module

https://gerrit.wikimedia.org/r/581647

gerritbot added a comment.Mar 20 2020, 2:52 PM

Change 581654 merged by Bstorm:
[operations/puppet@production] toolserver: refactor into profile and move off "toollabs" name

https://gerrit.wikimedia.org/r/581654

gerritbot added a comment.Mar 20 2020, 3:11 PM

Change 582065 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: remove the last toollabs role

https://gerrit.wikimedia.org/r/582065

• Bstorm updated the task description. (Show Details)Mar 20 2020, 3:18 PM
gerritbot added a comment.Mar 20 2020, 3:27 PM

Change 582065 merged by Bstorm:
[operations/puppet@production] toolforge: remove the last toollabs role

https://gerrit.wikimedia.org/r/582065

gerritbot added a comment.Mar 20 2020, 3:42 PM

Change 582090 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] k8s: purge flannel from the environment

https://gerrit.wikimedia.org/r/582090

• Bstorm added a comment.Mar 20 2020, 6:03 PM

This dashboard now works with the new cluster https://grafana-labs.wikimedia.org/d/000000005/kubernetes-worker-health?orgId=1&refresh=1m

• Bstorm added a comment.Mar 20 2020, 9:05 PM

I also shored up the etcd health dashboard.

• Bstorm added a comment.Mar 20 2020, 9:07 PM

It seems like the Kubernetes Cluster dash is totally useless. We have the Toolforge K8s dashboard, which is very nice so far (and newer!).
I think it's just an old one that we can delete https://grafana-labs.wikimedia.org/d/000000001/kubernetes-cluster?orgId=1
@aborrero does that sound right.

gerritbot added a comment.Mar 20 2020, 10:10 PM

Change 578408 merged by jenkins-bot:
[operations/software/tools-webservice@master] Remove temporary code from 2020 Kubernetes migration

https://gerrit.wikimedia.org/r/578408

aborrero added a comment.Mar 23 2020, 11:36 AM
In T246689#5988213, @Bstorm wrote:

It seems like the Kubernetes Cluster dash is totally useless. We have the Toolforge K8s dashboard, which is very nice so far (and newer!).
I think it's just an old one that we can delete https://grafana-labs.wikimedia.org/d/000000001/kubernetes-cluster?orgId=1
@aborrero does that sound right.

sounds right!

gerritbot added a comment.Mar 24 2020, 10:15 PM

Change 582090 merged by Bstorm:
[operations/puppet@production] k8s: purge flannel from the environment

https://gerrit.wikimedia.org/r/582090

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2020, 11:10 PM
gerritbot added a project: Patch-For-Review.Mar 24 2020, 11:18 PM
gerritbot added a comment.Mar 24 2020, 11:27 PM

Change 583165 merged by Bstorm:
[operations/puppet@production] toolforge: clean up the now-unneeded ferm_handlers

https://gerrit.wikimedia.org/r/583165

gerritbot added a comment.Mar 24 2020, 11:46 PM

Change 583170 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge cleanup: remove the ferm_handlers profile

https://gerrit.wikimedia.org/r/583170

gerritbot added a comment.Mar 25 2020, 5:30 PM

Change 583170 merged by Bstorm:
[operations/puppet@production] toolforge cleanup: remove the ferm_handlers profile

https://gerrit.wikimedia.org/r/583170

• Bstorm lowered the priority of this task from High to Medium.Mar 25 2020, 5:36 PM
• Bstorm updated the task description. (Show Details)

All high priority portions of this task are done. The rest is just cleanup that doesn't block anything else.

• Bstorm added a subtask: T248703: Rebuild the docker builder for Toolforge.Mar 27 2020, 8:19 PM
• Bstorm closed subtask T248703: Rebuild the docker builder for Toolforge as Resolved.Mar 27 2020, 9:30 PM
bd808 moved this task from WMCS infrastructure to Subtasks on the Cloud-VPS (Debian Jessie Deprecation) board.Mar 30 2020, 10:02 PM
• Bstorm added a comment.Mar 30 2020, 11:35 PM

I've deleted the Kubernetes Cluster dashboard (redundant at best, but also broken).

Stashbot added a comment.Mar 30 2020, 11:42 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-30T23:42:21Z] <bstorm_> deleted "Kubernetes Cluster" and "Kubernetes Performance" dashboards T246689

• Bstorm updated the task description. (Show Details)Mar 31 2020, 12:57 AM
• Bstorm added a comment.Mar 31 2020, 3:48 PM

Apparently, to remove docker tags/images from Docker Hub, which seems to use a very similar API to our registry, you'd just literally send DELETE requests against those URLs. Sounds like time to experiment!

bd808 updated the task description. (Show Details)Sep 23 2020, 7:45 PM
bd808 closed this task as Resolved.Sep 23 2020, 7:48 PM
bd808 assigned this task to • Bstorm.

Old images were cleaned up in late July: https://sal.toolforge.org/tools?p=0&q=bd808+AND+images&d=2020-07-24

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits