In T344507#9102068, @bd808 wrote:

This is really the same bug as T344289: Corrupt $HOME/.kube/config preventing use of Kubernetes for wikiquantos, wikiroupas, and possibly more tools. Apparently the corruption of those config files included different truncations of the extra line's path. Here it starts /project/ when the prior set discovered was ata/project. I think I can do another round of checking where I just look for any lines that are not key: value hash entries in the yaml.

I checked a known good $HOME/.kube/config and found that grep -v ':' $HOME/.kube/config does not match any lines. I used that to check for other corruptions as I did in T344289#9094586 with the other pattern:

$ ssh tools-nfs.svc.tools.eqiad1.wikimedia.cloud
$ cd /srv/tools/project
$ sudo grep -lv ':' */.kube/config | wc -l
1798

This is a much more pervasive problem than I originally assumed.

I started out fixing the files manually using vim: sudo vim $(sudo grep -lv ':' */.kube/config). I setup a search within vim to find lines that do not contain a colon: /\v^.+(^.*:.*$)@<!$. Then I could repeat the search with n to jump to the next likely corrupt line, dd to delete the line, and :w|bd to write the file to disk and move to the next buffer. I fixed about 150 files this way before deciding that I would die of boredom before finishing the queue.

The reason for manual editing was that it looked like the grep was finding a small number of files with a different cause of lines without an embedded :. I didn't want to blindly delete these lines with some sed or awk magic; instead I wanted to manually examine them.

After some thinking I decided to automate cleaning the general case error of the last line of the $HOME/.kube/config being substring of $HOME/.kube/.toolskube/client.key. Experimentation convinced me that sudo grep -v ':' */.kube/config | grep 'client' | cut -d: -f 1 was matching the set of files I wanted. Deleting the last line of a file can be done with sed --in-place '$d' <path>.

$ sudo grep -v ':' */.kube/config | grep 'client' | cut -d: -f 1 | wc -l
1502
$ sudo grep -v ':' */.kube/config | grep 'client' | cut -d: -f 1 | sudo xargs sed --in-place '$d'
$ sudo grep -v ':' */.kube/config | grep 'client' | cut -d: -f 1 | wc -l
0

After that I returned to manually editing the remaining files:

$ sudo vim $(sudo grep -lv ':' */.kube/config)
145 files to edit
$ sudo grep -lv ':' */.kube/config | wc -l
0

Most of these "extra" 145 files had a corrupt value for the auth token from the legacy (2016) Kubernetes cluster.