Page MenuHomePhabricator

worker nodes issue with garbage collection
Closed, DuplicatePublic

Description

T375988 was opened and some investigation found:

Warning  FreeDiskSpaceFailed  118s (x159 over 13h)  kubelet  Failed to garbage collect required amount of images. Attempted to free 4281512755 bytes, but only found 0 bytes eligible to free.

The worker nodes were replaced and that remedied the issue. Though it will likely return.

Doesn't immediately seem to be a return of T336586 as the control node did not appear to be impacted. If it has something to do with the upgrade to k8s 1.27 we will likely see the same in PAWS. Though perhaps not as fast as it has a larger disk.