Page MenuHomePhabricator

Reconfigure the Toolforge k8s workers to map their unused disk to /var/lib/docker
Closed, ResolvedPublic


The worker nodes in the Kubernetes cluster use the overlay2 storage driver. They therefore use /var/lib/docker/overlay2 for docker storage and such. Right now, that's all on a relatively smallish OS disk. That hasn't caused a huge problem because the disk is like 20G, but we have 60G unused that should be mounted at /var/lib/docker.

The tidiest fixup might be:

  1. Disable puppet across tools-k8s-worker-*
  2. Merge a patch that includes something like modules/profile/manifests/ci/dockervolume.pp
  3. drain the nodes one at a time and do the following to each:
    1. Stop docker
    2. Stop kubelet (just in case)
    3. Delete everything in /var/lib/docker/
    4. Enable puppet
    5. Run puppet until bleeding stops (if there is any)
    6. ensure docker and kubelet are running and happy
    7. uncordon node

Related Objects

Event Timeline

Bstorm triaged this task as High priority.Mar 27 2020, 6:09 PM
Bstorm created this task.

Honestly, this may be doable with a simple script and ssh for each node.

Change 584061 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: mount /var/lib/docker on appropriate volume

This worked great in toolsbeta 🙂

3set -Eeuo pipefail
5workers=(toolsbeta-test-k8s-worker-1.toolsbeta.eqiad.wmflabs toolsbeta-test-k8s-worker-2.toolsbeta.eqiad.wmflabs toolsbeta-test-k8s-worker-3.toolsbeta.eqiad.wmflabs)
8SSHCMD="/usr/bin/ssh -l root"
10for worker in ${workers[*]}
12 node=${worker%%.*}
13 echo "Draining $node"
14 $SSHCMD $controlnode "kubectl drain --force --ignore-daemonsets --delete-local-data $node"
15 echo "Stopping the services"
16 $SSHCMD "$worker" "systemctl stop docker"
17 $SSHCMD "$worker" "systemctl stop kubelet"
18 echo "destroying /var/lib/docker"
19 $SSHCMD "$worker" "rm -rf /var/lib/docker"
20 echo "bringing back puppet"
21 $SSHCMD "$worker" "puppet agent --enable"
22 $SSHCMD "$worker" "puppet agent -t" || true
23 $SSHCMD $controlnode "kubectl uncordon $node"
24 echo "***${node} is done***"

Mentioned in SAL (#wikimedia-cloud) [2020-03-30T18:22:32Z] <bstorm_> disabled puppet across tools-k8s-worker-[1-55].tools.eqiad.wmflabs T248702

Change 584061 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: mount /var/lib/docker on appropriate volume

Mentioned in SAL (#wikimedia-cloud) [2020-03-30T18:28:01Z] <bstorm_> Beginning rolling depool, remount, repool of k8s workers for T248702

5 nodes could not complete the process: tools-k8s-worker-[16-20].tools.eqiad.wmflabs. These are not set up for LVM and need a rebuild.

All other worker nodes are done.

Since those 5 nodes need a rebuild, I'll leave this open until that's done just as a reminder.

bd808 added a subscriber: bd808.

Since those 5 nodes need a rebuild, I'll leave this open until that's done just as a reminder.

I licked the rebuild cookie, so taking the task for now to remind me to do the needful. Current plan is to build +5 new worker nodes, pool them, and then drain and delete tools-k8s-worker-[16-20].tools.eqiad.wmflabs.

Mentioned in SAL (#wikimedia-cloud) [2020-05-05T21:51:01Z] <bd808> Building 5 new k8s worker nodes (T248702)

tools-k8s-worker-[56-60].tools.eqiad.wmflabs have been built and joined to the cluster. I'm going to let them run for a while before draining and deleting tools-k8s-worker-[16-20].tools.eqiad.wmflabs.

Mentioned in SAL (#wikimedia-cloud) [2020-05-06T17:56:29Z] <bd808> Cordoned tools-k8s-worker-[16-20] in preparation for decomm (T248702)

Mentioned in SAL (#wikimedia-cloud) [2020-05-06T18:04:10Z] <bd808> Draining tools-k8s-worker-[16-20] in preparation for decomm (T248702)

Mentioned in SAL (#wikimedia-cloud) [2020-05-06T18:14:24Z] <bd808> Shutdown tools-k8s-worker-[16-20] instances (T248702)

Mentioned in SAL (#wikimedia-cloud) [2020-05-06T18:24:23Z] <bd808> Updated "profile::toolforge::k8s::worker_nodes" list in "tools-k8s-haproxy" prefix puppet (T248702)

Mentioned in SAL (#wikimedia-cloud) [2020-05-06T21:20:47Z] <bd808> Kubectl delete node tools-k8s-worker-[16-20] (T248702)

bd808 reassigned this task from bd808 to Bstorm.