Page MenuHomePhabricator

maintain-kubeusers broken in Toolforge
Closed, ResolvedPublicBUG REPORT

Description

Was: webservice --backend=kubernetes python3.7 shell fails for new tool

maintain-kubeusers has begun crashing which will prevent new tools from getting Kubernetes credentials and stop renewal of credentials.

$ kubectl -n maintain-kubeusers get pods
NAME                                  READY   STATUS             RESTARTS   AGE
maintain-kubeusers-7f7b44754c-kkm76   0/1     CrashLoopBackOff   1513       32d

Event Timeline

Bstorm added a subscriber: Bstorm.

This suggests your tool does not have authentication credentials created. That either means you beat the service that creates that or that the service is broken.

Bstorm triaged this task as Unbreak Now! priority.Jan 12 2021, 6:11 PM
Bstorm moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

maintain-kubeusers-7f7b44754c-kkm76 0/1 CrashLoopBackOff 1513 32d
Unfortunately, the problem is the latter.

Bstorm renamed this task from webservice --backend=kubernetes python3.7 shell fails for new tool to maintain-kubeusers broken in Toolforge.Jan 12 2021, 6:13 PM
Bstorm updated the task description. (Show Details)
Bstorm updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-cloud) [2021-01-12T18:16:31Z] <bstorm> deleted wedged CSR tool-adhs-wde to get maintain-kubeusers working again T271842

$ kubectl -n maintain-kubeusers logs maintain-kubeusers-7f7b44754c-mgrjj
starting a run
Homedir already exists for /data/project/adhs-wde
Wrote config in /data/project/adhs-wde/.kube/config
Provisioned creds for user adhs-wde
finished run, wrote 1 new accounts

That fixed it. This was likely caused by a latency issue in etcd slowing down the cleanup of a failed request. Until we can make etcd more performant (T267966) we are going to see issues around that, so I think I need to teach this service how to clean up after itself (will create subtask).

tools.adhs-wde@tools-sgebastion-08:~$ webservice --backend=kubernetes python3.7 shell
Defaulting container name to interactive.
Use 'kubectl describe pod/interactive -n tool-adhs-wde' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.
tools.adhs-wde@interactive:~$

You should be good to go now. Thank you for the bug report!