List of steps to reproduce (step by step, including full links if applicable):
- On tools-sgebastion-10, as user tools.wikibugs
- toolforge-jobs load libera/k8s-jobs.yaml
What happens?:
ERROR: unable to create job: "HTTP 403: likely an internal bug: 403 Client Error: Forbidden for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/apis/apps/v1/namespaces/tool-wikibugs/deployments. k8s JSON: {\"kind\": \"Deployment\", \"apiVersion\": \"apps/v1\", \"metadata\": {\"name\": \"redis2irc\", \"namespace\": \"tool-wikibugs\", \"labels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}, \"spec\": {\"template\": {\"metadata\": {\"labels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}, \"spec\": {\"restartPolicy\": \"Always\", \"containers\": [{\"name\": \"redis2irc\", \"image\": \"docker-registry.tools.wmflabs.org/toolforge-python39-sssd-base:latest\", \"workingDir\": \"/data/project/wikibugs\", \"command\": [\"/bin/sh\", \"-c\", \"--\", \"libera/wrapper.sh redis2irc 1>>redis2irc.out 2>>redis2irc.err\"], \"resources\": {}, \"env\": [{\"name\": \"HOME\", \"value\": \"/data/project/wikibugs\"}], \"volumeMounts\": [{\"mountPath\": \"/data/project\", \"name\": \"home\"}]}], \"volumes\": [{\"name\": \"home\", \"hostPath\": {\"path\": \"/data/project\", \"type\": \"Directory\"}}]}}, \"replicas\": 1, \"selector\": {\"matchLabels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}}}"Afterwards, only two of the three expected jobs is running:
tools.wikibugs@tools-sgebastion-10:~$ toolforge-jobs list Job name: Job type: Status: ------------- ----------- --------- grrrrit continuous Running wikibugs-phab continuous Running
I've also attempted to flush and then load, but that doesn't seem to solve the issue.
What should have happened instead?:
All jobs should have started, no error should have been shown.
If an error had been shown, it should have been an error that is actionable rather than an incomprehensible json blurb.
Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc.: