Page MenuHomePhabricator

[infra] kubectl is quite slow the “first time” per user account
Open, LowPublic

Description

The “first time” you run kubectl in a certain user account (after “a while”, presumably some cache’s TTL), it’s quite slow to start up. Afterwards, it will be much faster in the same account, but still slow in another user account, even on the same host / bastion.

from 8½ seconds to half a second (2×)
lucaswerkmeister@tools-sgebastion-10:~$ time become lexeme-forms kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
lexeme-forms-f5794849b-jnvls   1/1     Running   0          10h

real    0m8,613s
user    0m0,324s
sys     0m0,358s
lucaswerkmeister@tools-sgebastion-10:~$ time become lexeme-forms kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
lexeme-forms-f5794849b-jnvls   1/1     Running   0          10h

real    0m0,586s
user    0m0,184s
sys     0m0,073s
lucaswerkmeister@tools-sgebastion-10:~$ time become lexeme-forms kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
lexeme-forms-f5794849b-jnvls   1/1     Running   0          10h

real    0m0,412s
user    0m0,178s
sys     0m0,089s
lucaswerkmeister@tools-sgebastion-10:~$ time become quickcategories kubectl get pods
NAME                                 READY   STATUS    RESTARTS      AGE
background-runner-688875655c-2k9jm   1/1     Running   6 (37h ago)   12d
quickcategories-686d67f74d-qddtc     1/1     Running   0             11d

real    0m8,353s
user    0m0,369s
sys     0m0,321s
lucaswerkmeister@tools-sgebastion-10:~$ time become quickcategories kubectl get pods
NAME                                 READY   STATUS    RESTARTS      AGE
background-runner-688875655c-2k9jm   1/1     Running   6 (37h ago)   12d
quickcategories-686d67f74d-qddtc     1/1     Running   0             11d

real    0m0,546s
user    0m0,186s
sys     0m0,085s
lucaswerkmeister@tools-sgebastion-10:~$ time become quickcategories kubectl get pods
NAME                                 READY   STATUS    RESTARTS      AGE
background-runner-688875655c-2k9jm   1/1     Running   6 (37h ago)   12d
quickcategories-686d67f74d-qddtc     1/1     Running   0             11d

real    0m0,424s
user    0m0,173s
sys     0m0,069s

I think to some extent this isn’t new, but it does feel like it’s been getting worse the past few weeks or so, and I thought it was worth mentioning here.

Event Timeline

Interestingly, this even seems to be the case right after a webservice restart, so it’s probably not (as I first thought it might be) slowness on the side of the k8s server when working in a previously-dormant namespace.

dcaro renamed this task from kubectl is quite slow the “first time” per user account to [infra] kubectl is quite slow the “first time” per user account.Mar 5 2024, 5:17 PM
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.