Page MenuHomePhabricator

install2002 short on disk space
Closed, ResolvedPublic

Description

There is a warning for install2002 on Icinga for being short on disk space on /

root@install2002:~# df -hT "/"
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vda1      ext4   78G   70G  4.1G  95% /

There was a big increase on disk the 6th August at around 12:40 UTC: https://grafana.wikimedia.org/render/d-solo/000000377/host-overview?refresh=5m&panelId=12&orgId=1&from=1565077487095&to=1565163887095&var-server=install2002&var-datasource=codfw%20prometheus%2Fops&var-cluster=misc&width=1000&height=500&tz=Europe%2FMadrid

root@install2002:/srv# du -sh .
65G	.
root@install2002:/srv# du -sh *
456K	autoinstall
6.8G	firmware
0	index.html
1.4M	ipmi
22G	junos
4.5M	megacli
4.0K	tftp
1.7G	tftpboot
35G	wikimedia

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2019, 7:46 AM
Marostegui triaged this task as Normal priority.Aug 7 2019, 7:48 AM
Marostegui added a project: Operations.
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-08-07T17:46:21Z] <shdubsh> install2002 stop nginx and squid for resync /srv to spare disk and restore mount - T229997

Mentioned in SAL (#wikimedia-operations) [2019-08-07T17:54:22Z] <shdubsh> install2002 add fstab entry for /srv mount - T229997

It looks like this has happened before, so I linked the case.

A reboot occurred and the disk that was added in December was not remounted. The installserver synchronization process filled /srv with replica data on the root disk because the other disk was not mounted.

I have remounted the disk to /srv and added an fstab entry to hopefully mitigate this problem in the future.

colewhite closed this task as Resolved.Aug 7 2019, 6:00 PM
Dzahn added a comment.Aug 8 2019, 11:22 PM

Oh, thanks. looks like i forgot to add it to fstab when i created and mounted it. This should fix it indeed.