kubernetes[2023-2024].codfw.wmnet,kubernetes[1023-1024].eqiad.wmnet are using devicemapper instead of overlay2 for some reason.

I've cleaned up some space so these messages won't re-appear for a bit, but we need to fix the underlying issue.

This is not a train blocker, as it only means we are not scheduling pods from new images on these hosts.

Thank you Clément for the quick assessment :]

Mentioned in SAL (#wikimedia-operations) [2023-03-22T20:32:42Z] <akosiaris> reboot kubernetes1023 for a test once more, ⚓ T332803

configuration looked fine, a drain, stop of docker, delete of /var/lib/docker and reboot fixed it for kubernetes1023. Now... why did it happened and just on those 2 nodes? Adding @elukey and @jijiki that imaged those nodes.

Mentioned in SAL (#wikimedia-operations) [2023-03-22T20:37:40Z] <akosiaris> uncordon reboot kubernetes1023. It was drained previously for ⚓ T332803

I have a theory about this. Role was applied to already imaged nodes, but without a full re-image. The end result was, docker started but the overlay module wasn't whitelisted by profile::base::overlayfs: true due to the lack of a reboot. Docker fell back to devicemapper. Pods got assigned and the /var/lib/docker directory got "locked" to being devicemapper even after the overlay module was enabled. Our configuration[1] is minimal and as far as the storage/graph driver goes, it sticks with defaults which supports this theory.

[1]

{
  "bridge": "none",
  "iptables": false,
  "ip-masq": false,
  "log-opts": {
    "max-size": "100m"
  },
  "live-restore": true
}

In T332803#8719449, @akosiaris wrote:

I have a theory about this. Role was applied to already imaged nodes, but without a full re-image. The end result was, docker started but the overlay module wasn't whitelisted by profile::base::overlayfs: true due to the lack of a reboot. Docker fell back to devicemapper. Pods got assigned and the /var/lib/docker directory got "locked" to being devicemapper even after the overlay module was enabled. Our configuration[1] is minimal and as far as the storage/graph driver goes, it sticks with defaults which supports this theory.

[1]

{
  "bridge": "none",
  "iptables": false,
  "ip-masq": false,
  "log-opts": {
    "max-size": "100m"
  },
  "live-restore": true
}

yep that sounds all correct. If you now have overlay2 supported, you should just get the node out of k8s rotation, stop docker, clean its whole directory, restart docker. That should fix it AFAIR.

Alex I think that your theory is correct, I definitely applied the role without rebooting IIRC, didn't check the docker settings. Should we have an alarm to identify this corner case?

Mentioned in SAL (#wikimedia-operations) [2023-03-23T07:42:43Z] <elukey> clean up docker on kubernetes1024 (cordon + stop kubelet + docker + clean /var/lib/docker/*) and reboot to enable overlay2 - T332803

Mentioned in SAL (#wikimedia-operations) [2023-03-23T07:54:35Z] <elukey> clean up docker and reboot kubernetes2023 to enable overlay2 - T332803

Mentioned in SAL (#wikimedia-operations) [2023-03-23T08:20:59Z] <elukey> clean up docker and reboot kubernetes2024 to enable overlay2 - T332803

All nodes should be running overlay now :)

Before closing - should we add an alarm for this use case?

In T332803#8720439, @elukey wrote:

All nodes should be running overlay now :)

Awesome thanks! Happy that my theory is probably correct :-)

Before closing - should we add an alarm for this use case?

What are we going to alert on though? Parsing docker info and alert if it using devicemapper? There's the following issues:

• For all of kubernetes we are forcibly moving away from Docker as past 1.24 it is not a supported Container Runtime Engine. So whatever we implement, will probably be docker specific and will be a wasted effort (for production at least) down the line.
• Devicemapper is a deprecated storage engine anyway and disabled in docker 23.x (see https://docs.docker.com/engine/deprecated/). It should not be used anyway in our environment.

It's quite possibly better to just force overlay2 in our config. If the config doesn't match the data directory, docker should fail to start (IIRC) and we 'll get an alert anyway then.

In T332803#8720455, @akosiaris wrote:

It's quite possibly better to just force overlay2 in our config. If the config doesn't match the data directory, docker should fail to start (IIRC) and we 'll get an alert anyway then.

That sounds good to me tbh. and easiest to implement as well.

Ack makes sense, let's do it :)

Change 902318 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Force docker storage-driver to overlay2

https://gerrit.wikimedia.org/r/902318

Change 902318 merged by JMeybohm:

[operations/puppet@production] k8s: Force docker storage-driver to overlay2

https://gerrit.wikimedia.org/r/902318

Change 903329 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] k8s: Use storage-driver instead of storage_driver

https://gerrit.wikimedia.org/r/903329

Change 903329 merged by JMeybohm:

[operations/puppet@production] k8s: Use storage-driver instead of storage_driver

https://gerrit.wikimedia.org/r/903329