Page MenuHomePhabricator

disk space on labvirt1007
Closed, ResolvedPublic

Description

labvirt1007 - Disk space

CRITICAL 2015-08-20 19:27:08 0d 8h 51m 46s 3/3 DISK CRITICAL - free space: /var/lib/nova/instances 28723 MB (1% inode=99%):

/dev/sdb1 2.2T 2.2T 29G 99% /var/lib/nova/instances

We'll need to find instances nobody uses and delete them or migrate some of them elsewhere.

Event Timeline

Dzahn raised the priority of this task from to Needs Triage.
Dzahn updated the task description. (Show Details)
Dzahn added projects: acl*sre-team, Cloud-VPS.
Dzahn added subscribers: Dzahn, Andrew.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald Transcript
Dzahn set Security to None.

Seems some disk space has been reclaimed on labvirt1007. I managed to boot the three CI instances that were paused (T110052).

@Andrew seems any instance on labvirt1007 might have ended up being corrupted. deployment-puppetmaster suffers from the same issue that occurred on the Jenkins slaves: files written too are created but are empty.

scfc added a subscriber: scfc.

Are the instances that are on this virtual host gone for good?

As far as I know, very little was permanently damaged by this. Certainly a random sampling of instances on labvirt1007 look fine to me. Hashar, any objection to my closing this?

hashar claimed this task.

Yup it is fine. File written too while the disk was full might end up with 0 bytes written too. For the CI slaves I just rebuild them to be safe.