Page MenuHomePhabricator

toolforge: Fix job/cronjob quotas
Closed, ResolvedPublicSecurity

Description

We currently provision quotas for jobs and cronjobs like this:

taavi@tools-sgebastion-10:~ $ kubectl describe quota -n tool-mismatch-finder-staging
Name:                   tool-mismatch-finder-staging
Namespace:              tool-mismatch-finder-staging
Resource                Used  Hard
--------                ----  ----
count/cronjobs          0     50
count/jobs              0     15
[...]

The quota resource names are wrong! According to https://kubernetes.io/docs/concepts/policy/resource-quotas/#object-count-quota, they should be count/jobs.batch and count/cronjobs.batch.

This just caused us some major issues when a misbehaving tool created an absolute ton of job objects all constantly trying to spawn a job but failing (hitting the pod limit).

TODO:

  • fix maintain-kubeusers for new tools
  • fix existing tools
  • double-check toolforge-jobs is setting a sensible concurrencyPolicy by default

Details

Author Affiliation
Wikimedia Communities

Event Timeline

taavi triaged this task as High priority.Feb 6 2022, 6:30 PM
taavi created this task.

Mentioned in SAL (#wikimedia-cloud) [2022-02-06T18:36:53Z] <taavi> manually add T301081 fixes to prevent further k8s cluster issues

Mentioned in SAL (#wikimedia-cloud) [2022-02-06T18:36:58Z] <taavi> manually add T301081 fixes to prevent further k8s cluster issues

taavi added a project: Kubernetes.

Created https://github.com/wmde/wikidata-mismatch-finder/pull/306 to fix the application causing issues this time. Although I'm not a fan of a cron job using * * * * *, at least it prevents a new pile-up of Job objects.

Thanks for the PR! We run this job each minute to delegate task scheduling to Laravel, as is specified in https://laravel.com/docs/8.x/scheduling#running-the-scheduler

If this is not a pattern we want to encourage, I'd recommend adding it alongside a mention of the concurrency policy setting to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_cronjobs in a place where it would be discoverable by any new tool creators that are not well versed in k8s.

Thanks for the PR! We run this job each minute to delegate task scheduling to Laravel, as is specified in https://laravel.com/docs/8.x/scheduling#running-the-scheduler

If this is not a pattern we want to encourage, I'd recommend adding it alongside a mention of the concurrency policy setting to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_cronjobs in a place where it would be discoverable by any new tool creators that are not well versed in k8s.

The main issue here is that unlike a traditional cron daemon running on a single server, running cron jobs on the Kubernetes cluster has a non-zero overhead since each execution needs to create and store the Job and Pod objects and their state, schedule the Pod on a suitable node, potentially pull down the docker image and so on. Using a cron-capable scheduler to run a job every minute to run another cron-capable scheduler is incredibly inefficient!

I wrote a quick proof of concept Laravel command (P20186) that can generate a YAML file using syntax that can be used with toolforge-jobs load. This lets you use the Kubernetes scheduler to run Artisan commands configured in app/Console/Kernel.php, which is much more efficient.

As a side note, I hope that the contents of https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes will be soon replaced with toolforge-jobs examples. The new jobs framework is currently in an open beta phase (T285944) and is designed to make it easy to create cron jobs and continuous deployments in the Kubernetes cluster with sensible defaults (like concurrencyPolicy: Forbid).

I wrote a quick proof of concept Laravel command (P20186) that can generate a YAML file using syntax that can be used with toolforge-jobs load. This lets you use the Kubernetes scheduler to run Artisan commands configured in app/Console/Kernel.php, which is much more efficient.

I like this proof of concept, I wonder where in the workflow this should be run. I suppose on every deployment to the toolforge instance.

As a side note, I hope that the contents of https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes will be soon replaced with toolforge-jobs examples. The new jobs framework is currently in an open beta phase (T285944) and is designed to make it easy to create cron jobs and continuous deployments in the Kubernetes cluster with sensible defaults (like concurrencyPolicy: Forbid).

That sounds exciting, looking forward to the more official launch.

Mentioned in SAL (#wikimedia-cloud) [2022-02-07T12:52:34Z] <taavi> updated maintain-kubeusers for T301081

Running the following script in a screen to fix existing quotas:

#!/bin/bash

set -Eeuo pipefail

declare -a namespaces
readarray -t namespaces < <(kubectl get quota -A | grep -v NAMESPACE | grep -v cronjobs.batch | awk '{print $1}')

for ns in "${namespaces[@]}"
do
    echo "Fixing ${ns}"
    kubectl get quota -n ${ns} -o json | jq '.items[0].spec.hard' | jq '.["count/jobs.batch"] = .["count/jobs"]' | jq '.["count/cronjobs.batch"] = .["count/cronjobs"]' | jq '{spec: {hard: .}}' | kubectl patch quota -n ${ns} ${ns} --patch "$(cat -)" --as admin --as-group system:masters
    kubectl patch quota -n ${ns} ${ns} --type json -p '[{"op": "remove", "path": "/spec/hard/count~1jobs"}, {"op": "remove", "path": "/spec/hard/count~1cronjobs"}]' --as admin --as-group system:masters
    sleep 1
done

echo "*********************"
echo "Done!"
taavi claimed this task.
Urbanecm changed the visibility from "Custom Policy" to "Public (No Login Required)".Feb 7 2022, 3:09 PM
Urbanecm changed the edit policy from "Custom Policy" to "All Users".