Mentioned in SAL (#wikimedia-cloud) [2022-02-06T18:36:53Z] <taavi> manually add T301081 fixes to prevent further k8s cluster issues

Mentioned in SAL (#wikimedia-cloud) [2022-02-06T18:36:58Z] <taavi> manually add T301081 fixes to prevent further k8s cluster issues

Created https://github.com/wmde/wikidata-mismatch-finder/pull/306 to fix the application causing issues this time. Although I'm not a fan of a cron job using * * * * *, at least it prevents a new pile-up of Job objects.

Thanks for the PR! We run this job each minute to delegate task scheduling to Laravel, as is specified in https://laravel.com/docs/8.x/scheduling#running-the-scheduler

If this is not a pattern we want to encourage, I'd recommend adding it alongside a mention of the concurrency policy setting to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_cronjobs in a place where it would be discoverable by any new tool creators that are not well versed in k8s.

In T301081#7688031, @ItamarWMDE wrote:

Thanks for the PR! We run this job each minute to delegate task scheduling to Laravel, as is specified in https://laravel.com/docs/8.x/scheduling#running-the-scheduler

If this is not a pattern we want to encourage, I'd recommend adding it alongside a mention of the concurrency policy setting to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_cronjobs in a place where it would be discoverable by any new tool creators that are not well versed in k8s.

The main issue here is that unlike a traditional cron daemon running on a single server, running cron jobs on the Kubernetes cluster has a non-zero overhead since each execution needs to create and store the Job and Pod objects and their state, schedule the Pod on a suitable node, potentially pull down the docker image and so on. Using a cron-capable scheduler to run a job every minute to run another cron-capable scheduler is incredibly inefficient!

I wrote a quick proof of concept Laravel command (P20186) that can generate a YAML file using syntax that can be used with toolforge-jobs load. This lets you use the Kubernetes scheduler to run Artisan commands configured in app/Console/Kernel.php, which is much more efficient.

As a side note, I hope that the contents of https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes will be soon replaced with toolforge-jobs examples. The new jobs framework is currently in an open beta phase (T285944) and is designed to make it easy to create cron jobs and continuous deployments in the Kubernetes cluster with sensible defaults (like concurrencyPolicy: Forbid).

I wrote a quick proof of concept Laravel command (P20186) that can generate a YAML file using syntax that can be used with toolforge-jobs load. This lets you use the Kubernetes scheduler to run Artisan commands configured in app/Console/Kernel.php, which is much more efficient.

I like this proof of concept, I wonder where in the workflow this should be run. I suppose on every deployment to the toolforge instance.

As a side note, I hope that the contents of https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes will be soon replaced with toolforge-jobs examples. The new jobs framework is currently in an open beta phase (T285944) and is designed to make it easy to create cron jobs and continuous deployments in the Kubernetes cluster with sensible defaults (like concurrencyPolicy: Forbid).

That sounds exciting, looking forward to the more official launch.

Mentioned in SAL (#wikimedia-cloud) [2022-02-07T12:52:34Z] <taavi> updated maintain-kubeusers for T301081

Running the following script in a screen to fix existing quotas:

#!/bin/bash

set -Eeuo pipefail

declare -a namespaces
readarray -t namespaces < <(kubectl get quota -A | grep -v NAMESPACE | grep -v cronjobs.batch | awk '{print $1}')

for ns in "${namespaces[@]}"
do
    echo "Fixing ${ns}"
    kubectl get quota -n ${ns} -o json | jq '.items[0].spec.hard' | jq '.["count/jobs.batch"] = .["count/jobs"]' | jq '.["count/cronjobs.batch"] = .["count/cronjobs"]' | jq '{spec: {hard: .}}' | kubectl patch quota -n ${ns} ${ns} --patch "$(cat -)" --as admin --as-group system:masters
    kubectl patch quota -n ${ns} ${ns} --type json -p '[{"op": "remove", "path": "/spec/hard/count~1jobs"}, {"op": "remove", "path": "/spec/hard/count~1cronjobs"}]' --as admin --as-group system:masters
    sleep 1
done

echo "*********************"
echo "Done!"