Page MenuHomePhabricator
Create Task
Maniphest T289131

Enable the Priority admission plugin
Closed, ResolvedPublic
Actions

Description

During docker stress testing we had multiple issues with kubernetes nodes (the ganeti ones) failing completely because calico-node was evicted from them. T289111 describes the aftermath of that exact situation (which stayed undetected mainly because eqiad is depooled).

Enabling the admission controller is the easy task but we will also want to limit the Kubernetes default priority classes system-cluster-critical and system-node-critical to only be used for Pods in namespaces we "trust" (like kube-system for services clusters, ml may need additional for istio, kf*).

This can be done by providing AdmissionConfiguration via kube-apiserver flag --admission-control-config-file like:

apiVersion: apiserver.k8s.io/v1alpha1
kind: AdmissionConfiguration
plugins:
- name: "ResourceQuota"
  configuration:
    apiVersion: resourcequota.admission.k8s.io/v1beta1
    kind: Configuration
    limitedResources:
    - resource: pods
      matchScopes:
      - scopeName: PriorityClass 
        operator: In
        values: 
        - system-cluster-critical
        - system-node-critical

And explicitly granting namespaces the permission to use those classes by adding a ResourceQuota object:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: priorityclass
  namespace: kube-system
spec:
  scopeSelector:
    matchExpressions:
      - operator : In
        scopeName: PriorityClass
        values: 
        - system-cluster-critical
        - system-node-critical

Relevant reads:

Details

SubjectRepoBranchLines +/-
role::ml_k8s::master: enable Priority pluginoperations/puppetproduction+0 -1
kubernetes: Limit use of PriorityClassoperations/puppetproduction+14 -0
kubernetes/staging: Limit use of PriorityClassoperations/puppetproduction+14 -14
kubernetes: Enable Priority admission pluginoperations/puppetproduction+0 -1
k8s::apiserver: Add admission controller config fileoperations/puppetproduction+55 -11
admin_ng: Deploy a ResourceQuota to allow priority pods in kube-systemoperations/deployment-chartsmaster+27 -1
kubernetes/staging: Enable Priority admission plugin in stagingoperations/puppetproduction+1 -8
kubernetes/staging: Enable Priority admission plugin in codfwoperations/puppetproduction+6 -0
kubernetes/staging: Reorder hiera keys to match production orderoperations/puppetproduction+7 -7
Customize query in gerrit

Related Objects

Event Timeline

JMeybohm triaged this task as High priority.Aug 18 2021, 11:07 AM
JMeybohm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 18 2021, 11:07 AM
JMeybohm mentioned this in T289111: kubernetes1005 BGP down for 3 weeks.Aug 18 2021, 11:08 AM
JMeybohm added a subscriber: elukey.
RhinosF1 subscribed.Aug 18 2021, 12:13 PM
gerritbot added a comment.Aug 19 2021, 7:17 AM

Change 713804 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes/staging: Reorder hiera keys to match production order

https://gerrit.wikimedia.org/r/713804

gerritbot added a project: Patch-For-Review.Aug 19 2021, 7:17 AM

Change 713805 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes/staging: Enable Priority admission plugin in codfw

https://gerrit.wikimedia.org/r/713805

gerritbot added a comment.Aug 19 2021, 7:17 AM

Change 713806 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes/staging: Enable Priority admission plugin in staging

https://gerrit.wikimedia.org/r/713806

gerritbot added a comment.Aug 19 2021, 7:17 AM

Change 713807 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes: Enable Priority admission plugin

https://gerrit.wikimedia.org/r/713807

JMeybohm claimed this task.Aug 19 2021, 7:19 AM
gerritbot added a comment.Aug 19 2021, 7:30 AM

Change 713804 merged by JMeybohm:

[operations/puppet@production] kubernetes/staging: Reorder hiera keys to match production order

https://gerrit.wikimedia.org/r/713804

gerritbot added a comment.Aug 19 2021, 7:30 AM

Change 713805 merged by JMeybohm:

[operations/puppet@production] kubernetes/staging: Enable Priority admission plugin in codfw

https://gerrit.wikimedia.org/r/713805

JMeybohm updated the task description. (Show Details)Aug 19 2021, 8:42 AM
JMeybohm updated the task description. (Show Details)
JMeybohm added subscribers: Jelto, akosiaris.
JMeybohm updated the task description. (Show Details)Aug 19 2021, 8:47 AM
JMeybohm added a comment.Aug 19 2021, 9:08 AM

After enabling the admission plugin in staging-codfw I and deleting Pods that do define a priorityClass, the priority is added correctly:

# kubectl -n kube-system describe po calico-node-7s266 |grep Priority
Priority Class Name:  system-node-critical
# kubectl -n kube-system describe po calico-node-ljj9b |grep Priority
Priority:             2000001000
Priority Class Name:  system-node-critical

Running pods are ofc. unaffected, pods (from sevices) not specifying priorityClass are assigned with "Priority: 0" as expected.

gerritbot added a comment.Aug 20 2021, 10:34 AM

Change 713806 merged by JMeybohm:

[operations/puppet@production] kubernetes/staging: Enable Priority admission plugin in staging

https://gerrit.wikimedia.org/r/713806

gerritbot added a comment.Aug 20 2021, 11:14 AM

Change 714038 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] admin_ng: Deploy a ResourceQuota to allow priority pods in kube-system

https://gerrit.wikimedia.org/r/714038

Stashbot added a comment.Aug 20 2021, 12:00 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-20T12:00:48Z] <jayme> enabled priority admission plugin on k8s staging, rolling restart all pods in kube-system namespace - T289131

Stashbot added a comment.Aug 20 2021, 3:37 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-20T15:37:19Z] <jayme> deleting various pods from staging to have them recreated with priorities - T289131

gerritbot added a comment.Aug 20 2021, 4:26 PM

Change 714071 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s::apiserver: Add admission controller config file

https://gerrit.wikimedia.org/r/714071

gerritbot added a comment.Aug 24 2021, 2:48 PM

Change 714038 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: Deploy a ResourceQuota to allow priority pods in kube-system

https://gerrit.wikimedia.org/r/714038

gerritbot added a comment.Aug 25 2021, 8:53 AM

Change 714071 merged by JMeybohm:

[operations/puppet@production] k8s::apiserver: Add admission controller config file

https://gerrit.wikimedia.org/r/714071

JMeybohm added a comment.Aug 25 2021, 9:32 AM

Test in staging-codfw looks as expected:

root@deploy1002:~# kubectl -n miscweb apply -f /home/jayme/debug_pod.yaml                                                                                
Error from server (Forbidden): error when creating "/home/jayme/debug_pod.yaml": pods "jayme" is forbidden: pods with system-cluster-critical priorityClass is not permitted in miscweb namespace
root@deploy1002:~# kubectl -n kube-system apply -f /home/jayme/debug_pod.yaml 
pod/jayme created
root@deploy1002:~# kubectl -n kube-system delete po jayme
pod "jayme" deleted
root@deploy1002:~# sed 's/system-cluster-critical/system-node-critical/' -i /home/jayme/debug_pod.yaml
root@deploy1002:~# kubectl -n miscweb apply -f /home/jayme/debug_pod.yaml 
Error from server (Forbidden): error when creating "/home/jayme/debug_pod.yaml": pods "jayme" is forbidden: pods with system-node-critical priorityClass is not permitted in miscweb namespace
root@deploy1002:~# 
root@deploy1002:~# kubectl -n kube-system apply -f /home/jayme/debug_pod.yaml 
pod/jayme created
root@deploy1002:~# kubectl -n kube-system delete po jayme
pod "jayme" deleted
gerritbot added a comment.Aug 25 2021, 9:35 AM

Change 713807 merged by JMeybohm:

[operations/puppet@production] kubernetes: Enable Priority admission plugin

https://gerrit.wikimedia.org/r/713807

gerritbot added a comment.Aug 25 2021, 9:44 AM

Change 714717 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes/staging: Limit use of PriorityClass

https://gerrit.wikimedia.org/r/714717

gerritbot added a comment.Aug 25 2021, 9:44 AM

Change 714718 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes: Limit use of PriorityClass

https://gerrit.wikimedia.org/r/714718

gerritbot added a comment.Aug 25 2021, 9:45 AM

Change 714717 merged by JMeybohm:

[operations/puppet@production] kubernetes/staging: Limit use of PriorityClass

https://gerrit.wikimedia.org/r/714717

gerritbot added a comment.Aug 25 2021, 11:34 AM

Change 714718 merged by JMeybohm:

[operations/puppet@production] kubernetes: Limit use of PriorityClass

https://gerrit.wikimedia.org/r/714718

Stashbot added a comment.Aug 25 2021, 11:39 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-25T11:39:13Z] <jayme> slowly restarting all pods in kube-system namespace in eqiad k8s cluster - T289131

Stashbot added a comment.Aug 25 2021, 1:02 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-25T13:02:58Z] <jayme> restarted all pods in kube-system namespace in codfw k8s cluster - T289131

JMeybohm closed this task as Resolved.Aug 30 2021, 9:13 AM

This is done

gerritbot added a comment.Feb 16 2022, 4:58 PM

Change 763277 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::ml_k8s::master: enable Priority plugin

https://gerrit.wikimedia.org/r/763277

gerritbot added a comment.Feb 17 2022, 9:28 AM

Change 763277 merged by Elukey:

[operations/puppet@production] role::ml_k8s::master: enable Priority plugin

https://gerrit.wikimedia.org/r/763277

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL