Page MenuHomePhabricator

webservice 0.69+ fills /tmp with k8s ca cert files
Open, HighPublic

Description

The main Toolforge bastion alerted today due to disk space on /. I expected that was apt filling things up again, but apt clean didn't get me as low as expected (still 16GB used). While this may not be the total fix of that situation, one problem is 3.3GB in use on /tmp, and there are 2051 temporary cert files in /tmp that are contributing to that based on sudo grep -o 'BEGIN' tmp* 2>/dev/null | wc -l in that directory.

I'm cleaning up the files manually, but we need to fix webservice so it either cleans up the file or (better yet) doesn't need to write it out to use the CA information.

Event Timeline

Bstorm triaged this task as High priority.Fri, May 22, 7:02 PM
Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, May 22, 7:02 PM

Mentioned in SAL (#wikimedia-cloud) [2020-05-22T19:12:08Z] <bstorm_> running command to delete over 2000 tmp ca certs on tools-bastion-07 T253412

That is definitely not what is filling the disk on the bastion at least!

Change 598109 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Remove validation of Kubernetes self-signed API cert

https://gerrit.wikimedia.org/r/598109