Log files filling up k8s worker nodes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	yuvipanda
	Oct 18 2016, 1:22 AM

Description

Docker logs are stored in /var/lib, and that is on the root partition with 19G of space only. It fills up easily when something is logging too much into stdout (jupyterhub was this time, I've been fixing it).

Related Objects

Mentioned In: T284656: Toolforge k8s: Migrate workers to Containerd and Bookworm

Event Timeline

yuvipanda created this task.Oct 18 2016, 1:22 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 18 2016, 1:22 AM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:40 PM

bd808 edited projects, added Toolforge; removed Cloud-Services.Jan 4 2020, 10:33 PM

Is this "instance's extra storage should be mounted at /var/lib" or "get people to log less stuff"? Or somehow send logs to NFS?

I think this is something that @Bstorm and/or @aborrero fixed with the 2020 cluster design? I certainly remember @Bstorm mentioning it as a concern.

This task was partially fixed way before the 2020 cluster (and before me). Then I fixed it the rest of the way on both the old and the new cluster. We limit docker log sizes via the native docker config on the new cluster.

• Bstorm mentioned this in T284656: Toolforge k8s: Migrate workers to Containerd and Bookworm.Oct 12 2021, 3:35 PM

Log files filling up k8s worker nodesClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Log files filling up k8s worker nodes
Closed, ResolvedPublic
Actions