Disk full on deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dancy
	May 5 2026, 2:54 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

dancy

Description

I got a notice that puppet is failing to run on deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud. I logged in to take a look and found a full root filesystem:

# df -t ext4 -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G   20G     0 100% /

Biggest consumers:

# du -x -h / | sort -rh | head
20G     /
18G     /var
12G     /var/log
4.8G    /var/lib
4.6G    /var/lib/docker
4.1G    /var/log/journal
3.7G    /var/lib/docker/containers
3.3G    /var/lib/docker/containers/3c62560a4f3ee6053036c5864832f0f6276e1e7da66e1d4f8d565f0b07eb1d8f
2.2G    /usr
2.1G    /var/log/journal/8b88deadf9ad4fff9ec45b52425de8ac

Event Timeline

dancy created this task.May 5 2026, 2:54 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2026, 2:54 PM

Mentioned in SAL (#wikimedia-releng) [2026-05-05T14:58:42Z] <dancy> rm /var/log/{user.log.1,syslog.1,messages.1} on deployment-eventgate-4.deployment- prep (T425429)

After deleting some old system log files:

# df -h -t ext4
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G   12G  6.8G  64% /

taavi added a project: Beta-Cluster-Infrastructure.May 5 2026, 3:00 PM

I also ran journalctl --vacuum-size=1G to further trim logs.

# df -h -t ext4
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G  7.3G   12G  39% /

And I rebooted in case any programs crashed due to the full filesystem. The puppet agent is working now.

Disk full on deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloudClosed, ResolvedPublicActions

Description

Event Timeline

Disk full on deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud
Closed, ResolvedPublic
Actions