thanos-be2004's filesystem on sdb3 is reported as basically full:
/dev/sdb3 94G 92G 1.3G 99% /srv/swift-storage/sdb3
We have big containers at the moment due to tegola's usage (e.g. T307184: Followups for Tegola and Swift interactions ), and the lack of space is exacerbated by quarantined databases in this case (due to container-replicator failures)
# du -hcs /srv/swift-storage/sdb3/* 1.0M /srv/swift-storage/sdb3/accounts 56G /srv/swift-storage/sdb3/containers 25G /srv/swift-storage/sdb3/quarantined 13G /srv/swift-storage/sdb3/tmp 92G total
Growing SSD container partition
The solution I have devised is to grow sda/sdb 3 partitions: the swift-grow-ssd-part script will delete the 4 partition, append space at the end of 3 partition and recreate the 4 partition. On the filesystem end it will xfs_grow the 3 partition online and mkfs.xfs the 4 partition. We lose the filesystem on the 4 partition but that's okay, data will be reconstructed by swift (and the 4 partition is largely unused anyways).
The procedure is as follows:
### The script will need the --doit flag to act on the disk! swift-grow-ssd-part --amount 100G --dev /dev/sda swift-grow-ssd-part --amount 100G --dev /dev/sdb mount -a run-puppet-agent