We've noticed that when we try to extract the mediawiki image onto old k8s servers with spinning disks, it often times out.
In order to allow us to do this, we might need to add a new annotation to the nodes.
Description
Description
Details
Details
Customize query in gerrit
Related Objects
Related Objects
- Mentioned In
- T289834: Add network policies to the ML k8s clusters
T280497: Benchmark performance of MediaWiki on k8s
T284628: Kubernetes timeing out before pulling the mediawiki-multiversion image
T288509: Add a fact holding the type of a disk (spinning/ssd) - Mentioned Here
- T288509: Add a fact holding the type of a disk (spinning/ssd)
Event Timeline
Comment Actions
For now I could limit mwdebug to run on physical nodes with node number higher than 4, but I'd think we might want this annotation for future use too
Comment Actions
We could maybe use the well known annotation node.kubernetes.io/instance-type [1] for this and come up with some type definitions for vm's and hardware nodes. It would be cool if puppet could populate that automatically based on facts.
[1] https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type
Comment Actions
Change 710566 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/puppet@production] kubernetes::node: Add node.kubernetes.io/disk-type annotation
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2021-08-09T11:50:05Z] <jayme> disabling puppet on all kubernetes nodes - T288345
Comment Actions
Change 710566 merged by JMeybohm:
[operations/puppet@production] kubernetes::node: Add node.kubernetes.io/disk-type annotation
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2021-08-09T11:53:24Z] <jayme> running puppet on kubernetes staging nodes (-b1 -s10) - T288345
Comment Actions
Change 710965 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):
[operations/deployment-charts@master] mwdebug: only deploy to nodes with ssd disks
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2021-08-09T14:06:53Z] <jayme> re-enabled (and ran) puppet on all kubernetes nodes - T288345
Comment Actions
That was a bit of a pain....
Anyways: All nodes do have proper node.kubernetes.io/disk-type labels now.
Comment Actions
Change 710965 merged by jenkins-bot:
[operations/deployment-charts@master] mwdebug: only deploy to nodes with ssd disks
Comment Actions
This is done for mediawiki pods now. For improvement of the current solution I did create T288509
Comment Actions
Change 711455 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/puppet@production] kubernetes:node: Fix disk-type annotation
Comment Actions
Change 711455 merged by JMeybohm:
[operations/puppet@production] kubernetes:node: Fix disk-type annotation
Comment Actions
Change 714962 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/puppet@production] kubernetes::node: Make use of the disk_type face
Comment Actions
Change 714962 merged by JMeybohm:
[operations/puppet@production] kubernetes::node: Make use of the disk_type fact
Comment Actions
To keep archives happy - this was done manually via kubectl, since the labels that we assign through the kubelet are evaluated only at node registration time.
Comment Actions
Change 754960 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] kuberenetes: disable mwautopull timer
Comment Actions
Change 754960 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] kuberenetes: disable mwautopull timer
Comment Actions
Change 754960 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] kuberenetes: disable mwautopull timer