Page MenuHomePhabricator
Create Task
Maniphest T408707

[jobs-api] apply topology constraints
Closed, ResolvedPublicFeature
Actions

Description

A job can be specified with multiple replicas.

Once https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/220 is merged with the default rollout strategy of

strategy:
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
  type: RollingUpdate

There is no good reason why jobs (deployments) should have downtime (bar some major platform problem).

One last shoot-yourself-in-the-face issue is we currently do not apply any constraints to the deployment, depending on resource usages at the time, all your replicas could get scheduled onto one node.

Additionally, as NFS nodes are not un-scheduleable for toolforge.org/mount-storage: none (like non nfs workers are for toolforge.org/mount-storage: all) it's also possible (perhaps likely due to the worker distribution) to get scheduled onto a node which is a little explody (nfs hangs).

To solve the first problem, we can easily apply topologySpreadConstraints to deployments.

For the second problem, today we apply (when mount=all):

nodeSelector:
  kubernetes.wmcloud.org/nfs-mounted: "true"

We could do the inverse, which would leave nfs workers with /only/ nfs workloads. Personally I think that would be quite nice, a lot less state and potential issues, but it might drastically change capacity planning etc.

A middle ground would be to construct some rules that motivate the scheduler to not use nodes with that selector, but still allow it, providing an incremental path to scale down nfs workers (if this is a goal).

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
runtime - configure topologySpreadConstraintsrepos/cloud/toolforge/jobs-api!235damianfeature/spread-pods-across-nodesmain
Customize query in GitLab

Event Timeline

DamianZaremba created this task.Oct 29 2025, 3:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 29 2025, 3:18 PM
DamianZaremba added a project: Toolforge.Oct 29 2025, 3:19 PM
DamianZaremba updated the task description. (Show Details)
Restricted Application added a project: cloud-services-team. · View Herald TranscriptOct 29 2025, 3:20 PM
CodeReviewBot added a project: Patch-For-Review.Oct 29 2025, 5:35 PM

damian opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235

Draft: runtime - configure topologySpreadConstraints

CodeReviewBot added a comment.Oct 29 2025, 5:47 PM

damian updated https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38

Draft: Apply nodeAffinity when mount=none

fgiunchedi triaged this task as Medium priority.Oct 30 2025, 10:04 AM
dcaro subscribed.Oct 30 2025, 10:16 AM

Note that we have not had any worker get stuck anymore since the last OS upgrade of the NFS servers, so the differentiation of nfs/non-nfs might be less relevant in the near future (:fingerscrossed:).

DamianZaremba added a comment.Oct 30 2025, 1:13 PM
In T408707#11326613, @dcaro wrote:

Note that we have not had any worker get stuck anymore since the last OS upgrade of the NFS servers, so the differentiation of nfs/non-nfs might be less relevant in the near future (:fingerscrossed:).

That's good news.

I still think the volume-admission change is worthwhile from the aspects of leaving capacity for tools requiring nfs, motivation to move towards more stateless workers aside.

The jobs-api change is the more important bit.

CodeReviewBot added a comment.Nov 4 2025, 3:05 AM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235

runtime - configure topologySpreadConstraints

CodeReviewBot added a comment.Nov 4 2025, 3:08 AM

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1054

jobs-api: bump to 0.0.436-20251104030529-de6dcb36

CodeReviewBot added a comment.Nov 4 2025, 3:33 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1054

jobs-api: bump to 0.0.436-20251104030529-de6dcb36

CodeReviewBot added a comment.Nov 12 2025, 10:21 AM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38

Apply nodeAffinity when mount=none

DamianZaremba added a comment.Nov 13 2025, 3:00 PM

Looks like this is working as expected;

tools.cluebot3@tools-bastion-15:~$ kubectl get pods -o json | jq -r '.items[] | [.metadata.labels."app.kubernetes.io/name", .spec.nodeName] | join(": ")'
cluebot3: tools-k8s-worker-nfs-35
toolforge jobs restart cluebot3
tools.cluebot3@tools-bastion-15:~$ kubectl get pods -o json | jq -r '.items[] | [.metadata.labels."app.kubernetes.io/name", .spec.nodeName] | join(": ")'
cluebot3: tools-k8s-worker-113
tools.cluebot3@tools-bastion-15:~$ kubectl get pod cluebot3-766f9864d8-zqwhl -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    app.kubernetes.io/restartedAt: "2025-11-13T14:53:54.089886+00:00"
    cni.projectcalico.org/containerID: a735d51d14f6224d96ea9bfeb67cc251073086cdd6e073ae8de4e5ad3b1a22a4
    cni.projectcalico.org/podIP: 192.168.103.241/32
    cni.projectcalico.org/podIPs: 192.168.103.241/32
  creationTimestamp: "2025-11-13T14:54:10Z"
  generateName: cluebot3-766f9864d8-
  labels:
    app.kubernetes.io/component: deployments
    app.kubernetes.io/created-by: cluebot3
    app.kubernetes.io/managed-by: toolforge-jobs-framework
    app.kubernetes.io/name: cluebot3
    app.kubernetes.io/version: "2"
    jobs.toolforge.org/emails: none
    pod-template-hash: 766f9864d8
    toolforge: tool
    toolforge.org/mount-storage: none
  name: cluebot3-766f9864d8-zqwhl
  namespace: tool-cluebot3
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: cluebot3-766f9864d8
    uid: a80ed41e-7ae1-4526-8f4a-4e79ebc48e76
  resourceVersion: "3858219421"
  uid: ade3160c-3ffd-423d-bffb-3dddca5f233b
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: kubernetes.wmcloud.org/nfs-mounted
            operator: NotIn
            values:
            - "true"
        weight: 100
  containers:
  - command:
    - /bin/sh
    - -c
    - --
    - launcher run-bot
    env:
    - name: NO_HOME
      value: a buildservice pod does not need a home env
    - name: CLUEBOT3_BOT_PASSWORD
      valueFrom:
        secretKeyRef:
          key: CLUEBOT3_BOT_PASSWORD
          name: toolforge.envvar.v1.cluebot3-bot-password
    - name: TOOL_DEPLOY_TOKEN
      valueFrom:
        secretKeyRef:
          key: TOOL_DEPLOY_TOKEN
          name: toolforge.envvar.v1.tool-deploy-token
    - name: TOOL_REPLICA_PASSWORD
      valueFrom:
        secretKeyRef:
          key: TOOL_REPLICA_PASSWORD
          name: toolforge.envvar.v1.tool-replica-password
    - name: TOOL_REPLICA_USER
      valueFrom:
        secretKeyRef:
          key: TOOL_REPLICA_USER
          name: toolforge.envvar.v1.tool-replica-user
    - name: TOOL_TOOLSDB_PASSWORD
      valueFrom:
        secretKeyRef:
          key: TOOL_TOOLSDB_PASSWORD
          name: toolforge.envvar.v1.tool-toolsdb-password
    - name: TOOL_TOOLSDB_USER
      valueFrom:
        secretKeyRef:
          key: TOOL_TOOLSDB_USER
          name: toolforge.envvar.v1.tool-toolsdb-user
    - name: TOOL_TOOLFORGE_API_URL
      value: https://api.svc.tools.eqiad1.wikimedia.cloud:30003
    - name: TOOL_REDIS_URI
      value: redis://redis.svc.tools.eqiad1.wikimedia.cloud:6379
    - name: TOOL_ELASTICSEARCH_URL
      value: http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud:80
    image: tools-harbor.wmcloud.org/tool-cluebot3/cluebot3:latest
    imagePullPolicy: Always
    livenessProbe:
      exec:
        command:
        - /bin/sh
        - -c
        - health-check
      failureThreshold: 3
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: job
    resources:
      limits:
        cpu: "3"
        memory: 1Gi
      requests:
        cpu: "3"
        memory: "536870912"
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: false
      runAsGroup: 53114
      runAsNonRoot: true
      runAsUser: 53114
    startupProbe:
      exec:
        command:
        - /bin/sh
        - -c
        - health-check
      failureThreshold: 120
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 5
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-2zmvm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: tools-k8s-worker-113
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 53114
    runAsGroup: 53114
    runAsNonRoot: true
    runAsUser: 53114
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 15
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-2zmvm
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-11-13T14:56:25Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-11-13T14:54:10Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-11-13T14:56:27Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-11-13T14:56:27Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-11-13T14:54:10Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://18493bf6f17fbb45d0325ff70546e070b8fd8724639626b97d9f0b929934dc25
    image: tools-harbor.wmcloud.org/tool-cluebot3/cluebot3:latest
    imageID: tools-harbor.wmcloud.org/tool-cluebot3/cluebot3@sha256:4c44aa171c384aded0183cc2609da6cba0338d50be9336d49b7aba3a38726326
    lastState: {}
    name: job
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2025-11-13T14:56:25Z"
  hostIP: 172.16.17.52
  hostIPs:
  - ip: 172.16.17.52
  phase: Running
  podIP: 192.168.103.241
  podIPs:
  - ip: 192.168.103.241
  qosClass: Burstable
  startTime: "2025-11-13T14:54:10Z"
DamianZaremba closed this task as Resolved.Nov 13 2025, 3:19 PM
DamianZaremba claimed this task.
dcaro edited projects, added: Toolforge (Toolforge iteration 25); removed: Toolforge.Nov 13 2025, 3:54 PM
dcaro moved this task from Next Up to Done on the Toolforge (Toolforge iteration 25) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits