Page MenuHomePhabricator

webservice 0.69+ fills /tmp with k8s ca cert files
Closed, ResolvedPublic

Description

The main Toolforge bastion alerted today due to disk space on /. I expected that was apt filling things up again, but apt clean didn't get me as low as expected (still 16GB used). While this may not be the total fix of that situation, one problem is 3.3GB in use on /tmp, and there are 2051 temporary cert files in /tmp that are contributing to that based on sudo grep -o 'BEGIN' tmp* 2>/dev/null | wc -l in that directory.

I'm cleaning up the files manually, but we need to fix webservice so it either cleans up the file or (better yet) doesn't need to write it out to use the CA information.

Event Timeline

Bstorm created this task.

Mentioned in SAL (#wikimedia-cloud) [2020-05-22T19:12:08Z] <bstorm_> running command to delete over 2000 tmp ca certs on tools-bastion-07 T253412

That is definitely not what is filling the disk on the bastion at least!

Change 598109 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Remove validation of Kubernetes self-signed API cert

https://gerrit.wikimedia.org/r/598109

Change 598109 merged by jenkins-bot:
[operations/software/tools-webservice@master] Remove validation of Kubernetes self-signed API cert

https://gerrit.wikimedia.org/r/598109

Mentioned in SAL (#wikimedia-cloud) [2020-06-15T18:14:25Z] <bd808> Rebuilding all Docker images to pick up webservice 0.71 (T254640, T253412)

$ sudo grep -o 'BEGIN' tmp* 2>/dev/null | wc -l
0