Page MenuHomePhabricator

install2002 short on disk space
Closed, ResolvedPublic


There is a warning for install2002 on Icinga for being short on disk space on /

root@install2002:~# df -hT "/"
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vda1      ext4   78G   70G  4.1G  95% /

There was a big increase on disk the 6th August at around 12:40 UTC:

root@install2002:/srv# du -sh .
65G	.
root@install2002:/srv# du -sh *
456K	autoinstall
6.8G	firmware
0	index.html
1.4M	ipmi
22G	junos
4.5M	megacli
4.0K	tftp
1.7G	tftpboot
35G	wikimedia

Related Objects

Event Timeline

Marostegui triaged this task as Medium priority.Aug 7 2019, 7:48 AM
Marostegui added a project: SRE.
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-08-07T17:46:21Z] <shdubsh> install2002 stop nginx and squid for resync /srv to spare disk and restore mount - T229997

Mentioned in SAL (#wikimedia-operations) [2019-08-07T17:54:22Z] <shdubsh> install2002 add fstab entry for /srv mount - T229997

It looks like this has happened before, so I linked the case.

A reboot occurred and the disk that was added in December was not remounted. The installserver synchronization process filled /srv with replica data on the root disk because the other disk was not mounted.

I have remounted the disk to /srv and added an fstab entry to hopefully mitigate this problem in the future.

Oh, thanks. looks like i forgot to add it to fstab when i created and mounted it. This should fix it indeed.