Page MenuHomePhabricator

disk space on labvirt1007
Closed, ResolvedPublic

Description

labvirt1007 - Disk space

CRITICAL 2015-08-20 19:27:08 0d 8h 51m 46s 3/3 DISK CRITICAL - free space: /var/lib/nova/instances 28723 MB (1% inode=99%):

/dev/sdb1 2.2T 2.2T 29G 99% /var/lib/nova/instances

We'll need to find instances nobody uses and delete them or migrate some of them elsewhere.

Event Timeline

Dzahn created this task.Aug 20 2015, 7:30 PM
Dzahn raised the priority of this task from to Needs Triage.
Dzahn updated the task description. (Show Details)
Dzahn added projects: acl*sre-team, Cloud-VPS.
Dzahn added subscribers: Dzahn, Andrew.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 20 2015, 7:30 PM
Restricted Application added subscribers: Matanya, Aklapper. · View Herald Transcript
Dzahn updated the task description. (Show Details)Aug 20 2015, 7:31 PM
Dzahn set Security to None.
hashar added a subscriber: hashar.Aug 25 2015, 8:51 AM

Seems some disk space has been reclaimed on labvirt1007. I managed to boot the three CI instances that were paused (T110052).

@Andrew seems any instance on labvirt1007 might have ended up being corrupted. deployment-puppetmaster suffers from the same issue that occurred on the Jenkins slaves: files written too are created but are empty.

scfc added a subscriber: scfc.

Are the instances that are on this virtual host gone for good?

As far as I know, very little was permanently damaged by this. Certainly a random sampling of instances on labvirt1007 look fine to me. Hashar, any objection to my closing this?

hashar closed this task as Resolved.Aug 27 2015, 6:24 PM
hashar claimed this task.

Yup it is fine. File written too while the disk was full might end up with 0 bytes written too. For the CI slaves I just rebuild them to be safe.