Page MenuHomePhabricator
Create Task
Maniphest T358203

[k8s] Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant
Closed, ResolvedPublic
Actions

Description

T355883: Create a pool of NFS-less Toolforge Kubernetes workers introduced a new type of workers. As the number of them is relatively low, and as most of our infrastructure components do not have NFS access, the risk of all of the pods in a given deployment ending up on the same node is higher than I'd like. For that reason(*) we should tell the Kubernetes scheduler to spread them to different nodes if possible.

Since Kubernetes 1.19 the best way to do this is with topologySpreadConstraints on the kubernetes.io/hostname field: https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/.

(*): It'd always been a good idea, but now the risk of this causing issues is much higher.

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
taavi triaged this task as Medium priority.Feb 22 2024, 11:19 AM
taavi mentioned this in T355883: Create a pool of NFS-less Toolforge Kubernetes workers.Feb 22 2024, 11:35 AM
taavi removed taavi as the assignee of this task.Feb 22 2024, 1:47 PM
dcaro renamed this task from Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant to [k8s] Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant.Mar 5 2024, 5:13 PM
dcaro edited projects, added Toolforge (Toolforge iteration 07); removed Toolforge (Toolforge iteration 06).
Raymond_Ndibe claimed this task.Apr 2 2024, 10:18 PM
CodeReviewBot added a project: Patch-For-Review.Apr 2 2024, 10:35 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/82

[builds-api] add topologySpreadConstraints to deployment

dcaro updated the task description. (Show Details)Apr 3 2024, 10:34 AM
dcaro edited projects, added Toolforge (Toolforge iteration 08); removed Toolforge (Toolforge iteration 07).Apr 3 2024, 4:16 PM
Slst2020 changed the task status from Open to In Progress.Apr 4 2024, 6:51 AM
Slst2020 moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 08) board.
Raymond_Ndibe moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 08) board.Apr 4 2024, 10:08 PM
dcaro edited projects, added Toolforge (Toolforge iteration 09); removed Toolforge (Toolforge iteration 08).Apr 16 2024, 2:25 PM
dcaro moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 09) board.
dcaro moved this task from In Review to In Progress on the Toolforge (Toolforge iteration 09) board.Apr 30 2024, 1:15 PM
dcaro edited projects, added Toolforge (Toolforge iteration 10); removed Toolforge (Toolforge iteration 09).May 22 2024, 4:40 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 10) board.
dcaro edited projects, added Toolforge (Toolforge iteration 11); removed Toolforge (Toolforge iteration 10).Jun 4 2024, 2:56 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 11) board.
dcaro edited projects, added Toolforge (Toolforge iteration 12); removed Toolforge (Toolforge iteration 11).Jul 2 2024, 2:00 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 12) board.
dcaro edited projects, added Toolforge (Toolforge iteration 13); removed Toolforge (Toolforge iteration 12).Jul 17 2024, 11:22 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 13) board.
dcaro edited projects, added Toolforge (Toolforge iteration 14); removed Toolforge (Toolforge iteration 13).Aug 5 2024, 7:39 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 14) board.
Raymond_Ndibe added a comment.EditedAug 15 2024, 2:13 AM

This has been open for some time now so we should probably close it. To test it here is what I plan to do:

For each target service in the description (builds-api, jobs-api, etc),

(1) pick a node from the list

* toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud
* toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud
* toolsbeta-test-k8s-worker-nfs-1.toolsbeta.eqiad1.wikimedia.cloud
* toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud

and configure the pods of the service to be scheduled on that node, verify that all the pods of the service are being scheduled on that node.

(2) add the topology-spread-constraint with nodeAffinityPolicy set to ignore. deploy and observe the the pods are now getting spread.

(3) remove the nodeAffinity/nodeSelector that was configured in step (1) ( also remove the nodeAffinityPolicy from the topology-spread-constraint configuration )

All the above means we'll have to deploy each service three times on toolsbeta alone to mark this task completed and ensure the target behavior is achieved and reliable, for that particular service.
Any other way to have less number of deployments for this?

dcaro subscribed.Aug 15 2024, 7:48 AM
In T358203#10065728, @Raymond_Ndibe wrote:

Any other way to have less number of deployments for this?

Deploying is easy xd, so three times is not a lot.

Now, we don't need to do the full test for each deployment, and we don't need to do it in toolsbeta, lima-kilo now has several workers running where you can play to make sure the affininty policy is working as expected.

Then we can do one full test with one of the deployment in toolsbeta, and just copy the config for the rest, wdyt?

Raymond_Ndibe added a comment.Aug 15 2024, 2:45 PM
In T358203#10066033, @dcaro wrote:
In T358203#10065728, @Raymond_Ndibe wrote:

Any other way to have less number of deployments for this?

Deploying is easy xd, so three times is not a lot.

Now, we don't need to do the full test for each deployment, and we don't need to do it in toolsbeta, lima-kilo now has several workers running where you can play to make sure the affininty policy is working as expected.

Then we can do one full test with one of the deployment in toolsbeta, and just copy the config for the rest, wdyt?

ok that sounds better

CodeReviewBot added a comment.Aug 15 2024, 3:31 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/481

[toolforge-deploy] DO_NOT_MERGE : increase builds-api replicas in local env

CodeReviewBot added a comment.Aug 15 2024, 3:40 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/108

Draft: [builds-api] DO_NOT_MERGE: schedule all pods on toolforge-worker

CodeReviewBot added a comment.Aug 16 2024, 3:30 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/45

Draft: [envvars-api] DO_NOT_MERGE: schedule all pods on toolforge-worker

CodeReviewBot added a comment.Aug 16 2024, 3:31 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46

[builds-api] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Aug 16 2024, 7:36 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/183

Draft: [lima-kilo] DO_NOT_MERGE: enable node inclusion policy feature gate

Raymond_Ndibe added a comment.Aug 16 2024, 9:03 AM

Tested on lima-kilo:
https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46#note_100236

CodeReviewBot added a comment.Aug 16 2024, 9:13 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/35

[api-gateway] add topologySpreadConstraints to deployment

Raymond_Ndibe updated the task description. (Show Details)Aug 16 2024, 9:21 AM
CodeReviewBot added a comment.Aug 16 2024, 9:25 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/10

[envvars-admission] add topologySpreadConstraints to deployment

Raymond_Ndibe updated the task description. (Show Details)Aug 16 2024, 9:26 AM
CodeReviewBot added a comment.Aug 16 2024, 9:36 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/15

[volume-admission] add topologySpreadConstraints to deployment

Raymond_Ndibe updated the task description. (Show Details)Aug 16 2024, 9:37 AM
CodeReviewBot added a comment.Aug 16 2024, 9:44 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/117

[jobs-api] add topologySpreadConstraints to deployment

Raymond_Ndibe updated the task description. (Show Details)Aug 16 2024, 9:45 AM
CodeReviewBot added a comment.Aug 16 2024, 10:14 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/58

[builds-builder] add topologySpreadConstraints to deployment

Raymond_Ndibe updated the task description. (Show Details)Aug 16 2024, 10:15 AM
CodeReviewBot added a comment.Aug 16 2024, 10:21 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/8

[ingress-admission] add topologySpreadConstraints to deployment

Raymond_Ndibe updated the task description. (Show Details)Aug 16 2024, 10:22 AM
CodeReviewBot added a comment.Aug 16 2024, 10:32 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/6

[jobs-emailer] add topologySpreadConstraints to deployment

Raymond_Ndibe updated the task description. (Show Details)Aug 16 2024, 10:32 AM
CodeReviewBot added a comment.Aug 16 2024, 10:39 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/11

[registry-admission] add topologySpreadConstraints to deployment

Raymond_Ndibe updated the task description. (Show Details)Aug 16 2024, 10:39 AM
Raymond_Ndibe added a comment.Aug 16 2024, 10:59 AM
In T358203#10068690, @Raymond_Ndibe wrote:

Tested on lima-kilo:
https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46#note_100236

Tested on lima-kilo

CodeReviewBot added a comment.Aug 16 2024, 11:00 AM

raymond-ndibe closed https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/108

Draft: [builds-api] DO_NOT_MERGE: schedule all pods on toolforge-worker

Raymond_Ndibe moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 14) board.Aug 16 2024, 12:02 PM
dcaro moved this task from In Review to In Progress on the Toolforge (Toolforge iteration 14) board.Aug 28 2024, 10:27 AM
CodeReviewBot added a comment.Sep 2 2024, 12:22 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/6

[jobs-emailer] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 12:22 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/8

[ingress-admission] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 12:23 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/10

[envvars-admission] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 12:25 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/499

jobs-emailer: bump to 0.0.43-20240902122251-1677ec5f

CodeReviewBot added a comment.Sep 2 2024, 12:26 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/500

ingress-admission: bump to 0.0.49-20240902122319-b0b4ce16

CodeReviewBot added a comment.Sep 2 2024, 12:27 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/501

volume-admission: bump to 0.0.54-20240902122354-18500992

CodeReviewBot added a comment.Sep 2 2024, 12:28 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502

envvars-admission: bump to 0.0.17-20240902122423-94921cad

CodeReviewBot added a comment.Sep 2 2024, 12:28 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/11

[registry-admission] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 12:29 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/58

[builds-builder] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 12:29 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/117

[jobs-api] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 12:29 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/35

[api-gateway] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 12:30 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/503

builds-builder: bump to 0.0.117-20240902122920-3b0529c7

CodeReviewBot added a comment.Sep 2 2024, 12:31 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504

api-gateway: bump to 0.0.44-20240902122941-a7ca33e8

CodeReviewBot added a comment.Sep 2 2024, 12:31 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/505

registry-admission: bump to 0.0.50-20240902122909-507a2f0d

CodeReviewBot added a comment.Sep 2 2024, 12:31 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/506

jobs-api: bump to 0.0.332-20240902122934-464b67dd

CodeReviewBot added a comment.Sep 2 2024, 12:47 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46

[envvars-api] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 12:51 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/507

envvars-api: bump to 0.0.60-20240902124757-342c4931

CodeReviewBot added a comment.Sep 2 2024, 12:57 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/82

[builds-api] add topologySpreadConstraints to deployment

CodeReviewBot added a comment.Sep 2 2024, 1:12 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/508

builds-api: bump to 0.0.170-20240902125720-e7b15322

CodeReviewBot added a comment.Sep 3 2024, 10:05 AM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/503

builds-builder: bump to 0.0.117-20240902122920-3b0529c7

CodeReviewBot added a comment.Sep 3 2024, 10:22 AM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/508

builds-api: bump to 0.0.170-20240902125720-e7b15322

CodeReviewBot added a comment.Sep 3 2024, 12:48 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/507

envvars-api: bump to 0.0.60-20240902124757-342c4931

CodeReviewBot added a comment.Sep 3 2024, 1:05 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/506

jobs-api: bump to 0.0.332-20240902122934-464b67dd

CodeReviewBot added a comment.Sep 3 2024, 2:11 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/505

registry-admission: bump to 0.0.50-20240902122909-507a2f0d

CodeReviewBot added a comment.Sep 3 2024, 7:21 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502

envvars-admission: bump to 0.0.17-20240902122423-94921cad

CodeReviewBot added a comment.Sep 3 2024, 7:36 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/501

volume-admission: bump to 0.0.54-20240902122354-18500992

CodeReviewBot added a comment.Sep 3 2024, 7:54 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/499

jobs-emailer: bump to 0.0.43-20240902122251-1677ec5f

CodeReviewBot added a comment.Sep 3 2024, 8:20 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/500

ingress-admission: bump to 0.0.49-20240902122319-b0b4ce16

CodeReviewBot added a comment.Sep 3 2024, 8:25 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504

api-gateway: bump to 0.0.44-20240902122941-a7ca33e8

Raymond_Ndibe closed this task as Resolved.Sep 3 2024, 8:45 PM
Raymond_Ndibe updated the task description. (Show Details)
Raymond_Ndibe moved this task from In Progress to Done on the Toolforge (Toolforge iteration 14) board.
CodeReviewBot added a comment.Sep 11 2024, 2:40 PM

raymond-ndibe closed https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/45

Draft: [envvars-api] DO_NOT_MERGE: schedule all pods on toolforge-worker

CodeReviewBot added a comment.Sep 11 2024, 2:41 PM

raymond-ndibe closed https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/183

Draft: [lima-kilo] DO_NOT_MERGE: enable node inclusion policy feature gate

CodeReviewBot added a comment.Sep 11 2024, 2:41 PM

raymond-ndibe closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/481

Draft: [toolforge-deploy] DO_NOT_MERGE : increase envvars-api replicas in local env

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits