Page MenuHomePhabricator
Create Task
Maniphest T325139

Relax nodeAffinity of sessionstore
Closed, ResolvedPublic
Actions

Description

As a follow up for T325056

We currently pin sessionstore to dedicated nodes for isolation:

nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
      - matchExpressions:
          - key: dedicated
            operator: In
            values:
              - kask

This means that sessionstore pods will become unshedulable of there is no dedicated kask node available.

We should relax the affinity from requiredDuringSchedulingIgnoredDuringExecution to preferredDuringSchedulingIgnoredDuringExecution to prefer a degradation of separation for some time over a outage. This should be combined with a proper alerting rule that fires when sessionstore gets scheduled on non-dedicated nodes to a human can intervene and clear the situation right away, restoring separation.

Details

SubjectRepoBranchLines +/-
Fix preferredDuringScheduling[...] change for sessionstoreoperations/deployment-chartsmaster+14 -12
Relax nodeAffinity of sessionstore podsoperations/deployment-chartsmaster+2 -2
Alert on sessionstore scheduling on non-dedicated k8s hostsoperations/alertsmaster+37 -0
Customize query in gerrit

Related Objects

Event Timeline

JMeybohm triaged this task as Medium priority.Dec 14 2022, 9:59 AM
JMeybohm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2022, 9:59 AM
JMeybohm moved this task from Incoming 🐫 to 🛎 Services & Oids on the serviceops board.Dec 14 2022, 10:03 AM
akosiaris added a project: SRE-Sprint-Week-Sustainability-March2023.Mar 20 2023, 11:25 AM
akosiaris moved this task from Backlog to Doing on the SRE-Sprint-Week-Sustainability-March2023 board.
eoghan claimed this task.Mar 21 2023, 12:06 PM
akosiaris subscribed.Mar 21 2023, 12:11 PM

Note that we also have taints on the dedicated to sessionstore nodes (albeit marked as kask, to avoid having other things scheduled on those nodes). This is actually tangentially related, and not really interfering with this task, but noting it down for completeness

gerritbot added a comment.Mar 21 2023, 12:36 PM

Change 901572 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/deployment-charts@master] Relax nodeAffinity of sessionstore pods

https://gerrit.wikimedia.org/r/901572

gerritbot added a project: Patch-For-Review.Mar 21 2023, 12:36 PM
gerritbot added a comment.Mar 22 2023, 11:06 AM

Change 902052 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/alerts@master] Alert on sessionstore scheduling on non-dedicated k8s hosts

https://gerrit.wikimedia.org/r/902052

gerritbot added a comment.Mar 22 2023, 1:31 PM

Change 902052 merged by jenkins-bot:

[operations/alerts@master] Alert on sessionstore scheduling on non-dedicated k8s hosts

https://gerrit.wikimedia.org/r/902052

gerritbot added a comment.Mar 22 2023, 3:15 PM

Change 901572 merged by EoghanGaffney:

[operations/deployment-charts@master] Relax nodeAffinity of sessionstore pods

https://gerrit.wikimedia.org/r/901572

Maintenance_bot removed a project: Patch-For-Review.Mar 22 2023, 3:31 PM
gerritbot added a comment.Mar 22 2023, 3:37 PM

Change 902114 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/deployment-charts@master] Fix preferredDuringScheduling[...] change for sessionstore

https://gerrit.wikimedia.org/r/902114

gerritbot added a project: Patch-For-Review.Mar 22 2023, 3:37 PM
gerritbot added a comment.Mar 22 2023, 4:15 PM

Change 902114 merged by EoghanGaffney:

[operations/deployment-charts@master] Fix preferredDuringScheduling[...] change for sessionstore

https://gerrit.wikimedia.org/r/902114

Maintenance_bot removed a project: Patch-For-Review.Mar 22 2023, 4:30 PM
eoghan added a comment.Mar 22 2023, 4:57 PM

We've deployed the change to relax the nodeAffinity setting, tomorrow morning we'll drain one of the nodes to test that the alert works correctly and we can close this ticket.

eoghan closed this task as Resolved.Mar 24 2023, 10:53 AM

image.png (334×894 px, 73 KB)

We have an alert to catch the condition where a pod gets scheduled on a non-dedicated host. Future work would be to create a task from this alert, rather than an alarm, but I think we can close this for now!

eoghan moved this task from Doing to Done on the SRE-Sprint-Week-Sustainability-March2023 board.Mar 24 2023, 10:54 AM
akosiaris awarded a token.Mar 24 2023, 1:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL