Page MenuHomePhabricator
Create Task
Maniphest T229997

install2002 short on disk space
Closed, ResolvedPublic
Actions

Description

There is a warning for install2002 on Icinga for being short on disk space on /

root@install2002:~# df -hT "/"
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vda1      ext4   78G   70G  4.1G  95% /

There was a big increase on disk the 6th August at around 12:40 UTC: https://grafana.wikimedia.org/render/d-solo/000000377/host-overview?refresh=5m&panelId=12&orgId=1&from=1565077487095&to=1565163887095&var-server=install2002&var-datasource=codfw%20prometheus%2Fops&var-cluster=misc&width=1000&height=500&tz=Europe%2FMadrid

root@install2002:/srv# du -sh .
65G	.
root@install2002:/srv# du -sh *
456K	autoinstall
6.8G	firmware
0	index.html
1.4M	ipmi
22G	junos
4.5M	megacli
4.0K	tftp
1.7G	tftpboot
35G	wikimedia

Related Objects
Search...

Event Timeline

Marostegui created this task.Aug 7 2019, 7:46 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2019, 7:46 AM
Marostegui triaged this task as Medium priority.Aug 7 2019, 7:48 AM
Marostegui added a project: SRE.
Marostegui updated the task description. (Show Details)
Marostegui added subscribers: MoritzMuehlenhoff, Dzahn, colewhite.
colewhite claimed this task.Aug 7 2019, 5:37 PM
colewhite added a parent task: T211850: install2002 94% disk usage on "/".
Stashbot added a comment.Aug 7 2019, 5:46 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-07T17:46:21Z] <shdubsh> install2002 stop nginx and squid for resync /srv to spare disk and restore mount - T229997

Stashbot added a comment.Aug 7 2019, 5:54 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-07T17:54:22Z] <shdubsh> install2002 add fstab entry for /srv mount - T229997

colewhite added a comment.Aug 7 2019, 5:59 PM

It looks like this has happened before, so I linked the case.

A reboot occurred and the disk that was added in December was not remounted. The installserver synchronization process filled /srv with replica data on the root disk because the other disk was not mounted.

I have remounted the disk to /srv and added an fstab entry to hopefully mitigate this problem in the future.

colewhite closed this task as Resolved.Aug 7 2019, 6:00 PM
Dzahn added a comment.Aug 8 2019, 11:22 PM

Oh, thanks. looks like i forgot to add it to fstab when i created and mounted it. This should fix it indeed.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL