You're out of inodes:

root@codesearch9:~# df -hi /srv
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda         5.0M  5.0M     0  100% /srv

I could have sworn that, bugs aside, we have genuinely increased the codesearch disk size several times before, but I can't seem to find any past task mentioning or describing it.

Previously:

• T408221 (about /, not /srv)
• T337263 (/srv full, due to duplication bug after switching git repos)
• T203538 (/srv full, mitigated by reducing git-depth in Hound)
• T188477 (/ full or rater, out of inodes. mitigated by moving Hound to /srv/ and increasing inodes)

From https://sal.toolforge.org/log/AWHkdPJWkkJ8OkTwiKCs (2018-03-02)

<legoktm> ran `mkfs.ext4 -T news /dev/mapper/vd-second--local--disk` (h/t bd808) to quintuple the number of inodes

There does not appear to be a trace of this in Puppet now, so I guess this has since been lost.

But, in any event, with only 11G of 79G left, inodes aside, we are genuinely close to being full, so maybe this time we do actually increase the volume size?

The instance codesearch9 is of type g4.cores4.ram8.disk20. The part that it has 80G and not just 20 and the name in /dev/mapper/vd-second--local--disk indicates that we already added a second virtual disk in the past.

While there is a "resize instance" button in Horizon, this allows for selecting another instance flavor. None of the flavors have disks over 20G though.

And this second local disk does not appear to be network storage using cinder.

It's not immediately obvious to me how this second local disk was created.

https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances#Extend_a_volume

Oh, yes, of course you are right. Not sure how I did not see it, thanks!

Yea, so..next step is the "quota exceeded". Let me create a request to increase it.

quota request created -> T411062

per:

https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_instances#Increase_quotas_for_projects

https://phabricator.wikimedia.org/project/profile/2880/

In T411047#11407583, @Krinkle wrote:

https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances#Extend_a_volume

Seems like the resizing option means downtime though. Would be nice if we could create a new larger volume and copy data over and avoid the resizefs part as well.

The command once found by bd808 and used by legoktm:

<Krinkle> <legoktm> ran `mkfs.ext4 -T news /dev/mapper/vd-second--local--disk` (h/t bd808) to quintuple the number of inodes

What the capital -T does here is define a "usage type" (as opposed to -t for file system type). This told it to optimize for many small files and have more inodes.

The usage type "news" specifically means "handling a very large number of extremely small files".

And yet.. we STILL ran out.

We talked about how to determine if the usage type "news" (aka "small") was already set because the system has likely been recreated since then.

"small" is a preset for blocksize and inode_ratio:

< somiaj> look at /etc/mke2fs.conf, all small is doing is setting blocksize = 1024, inode_ratio = 4096

We currently have "blocksize = 4096, inode_ratio = 16384" in that file. So we are NOT currently optimized for small files.

It does not seem possible though to run tune2fs and modify this on an existing file system / no way to add more inodes to existing ext4 fs.

So I think we should prefer to create a new volume, create a new file system, with -T small, and copy data over to it.

But can we connect more than one volume to an instance?

In T411047#11408054, @Dzahn wrote:

So I think we should prefer to create a new volume, create a new file system, with -T small, and copy data over to it.

But can we connect more than one volume to an instance?

We don't have to copy over data per-se. We can detach the old volume, attach the new one, and let Puppet provision it and start fresh.

Mentioned in SAL (#wikimedia-cloud) [2025-12-03T23:22:17Z] <mutante> - shut down instance codesearch9; extending volume "data2" to double its size T411728 T411047

• successfully resized /dev/sda to double its size (80 -> 160GB) in Horizon (possible after we got the project quota)

• remounted volume and ran resize2fs

• confirmed there are plenty of inodes again

` shutdown -h now

click "resize volume" in web UI
start instance
volume gets mounted automatically
resize2fs /dev/sda
mount -o remount /dev/sda

`

Mentioned in SAL (#wikimedia-cloud) [2025-12-05T18:35:18Z] <mutante> re-enabled puppet on codesearch9 - service up; has double disk space vs before incident - T411047 T411728