Page MenuHomePhabricator

Only schedule mediawiki pods on nodes with non-spinning disks
Closed, ResolvedPublic

Description

We've noticed that when we try to extract the mediawiki image onto old k8s servers with spinning disks, it often times out.

In order to allow us to do this, we might need to add a new annotation to the nodes.

Event Timeline

For now I could limit mwdebug to run on physical nodes with node number higher than 4, but I'd think we might want this annotation for future use too

We could maybe use the well known annotation node.kubernetes.io/instance-type [1] for this and come up with some type definitions for vm's and hardware nodes. It would be cool if puppet could populate that automatically based on facts.

[1] https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type

Change 710566 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes::node: Add node.kubernetes.io/disk-type annotation

https://gerrit.wikimedia.org/r/710566

Mentioned in SAL (#wikimedia-operations) [2021-08-09T11:50:05Z] <jayme> disabling puppet on all kubernetes nodes - T288345

Change 710566 merged by JMeybohm:

[operations/puppet@production] kubernetes::node: Add node.kubernetes.io/disk-type annotation

https://gerrit.wikimedia.org/r/710566

Mentioned in SAL (#wikimedia-operations) [2021-08-09T11:53:24Z] <jayme> running puppet on kubernetes staging nodes (-b1 -s10) - T288345

Change 710965 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] mwdebug: only deploy to nodes with ssd disks

https://gerrit.wikimedia.org/r/710965

Mentioned in SAL (#wikimedia-operations) [2021-08-09T14:06:53Z] <jayme> re-enabled (and ran) puppet on all kubernetes nodes - T288345

That was a bit of a pain....
Anyways: All nodes do have proper node.kubernetes.io/disk-type labels now.

Change 710965 merged by jenkins-bot:

[operations/deployment-charts@master] mwdebug: only deploy to nodes with ssd disks

https://gerrit.wikimedia.org/r/710965

JMeybohm claimed this task.

This is done for mediawiki pods now. For improvement of the current solution I did create T288509

Change 711455 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes:node: Fix disk-type annotation

https://gerrit.wikimedia.org/r/711455

Change 711455 merged by JMeybohm:

[operations/puppet@production] kubernetes:node: Fix disk-type annotation

https://gerrit.wikimedia.org/r/711455

Change 714962 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes::node: Make use of the disk_type face

https://gerrit.wikimedia.org/r/714962

Change 714962 merged by JMeybohm:

[operations/puppet@production] kubernetes::node: Make use of the disk_type fact

https://gerrit.wikimedia.org/r/714962