The underlying problem is in modules/toollabs/manifests/static.pp:

labs_lvm::volume { 'cdnjs-disk':
    mountat => '/srv',
    size    => '100%FREE'
}

This uses the standard space/inode ratio of mkfs.ext4. It should pass a mkfs_opt parameter with the needed options. Then creating new instances should fix the issue (inodes cannot be added to an existing file system).

Is is breaking updates, yes.

You say that the problem is 'This uses the standard space/inode ratio'. Is the standard ratio 'barely any inodes'? Or is cdnjs absurdly file-heavy?

So they are currently being served out of tools-web-static-xx or something - which have bigger disks :) but yes cdnjs is crazily inode heavy. Is this problem in the old nodes which I might not have deleted or the new ones?

(The standard space/inode ratio for ext4 is about 16384 bytes/inode.)

It's somewhat inode heavy, but not even that crazily. Based on du -aSb | cut -f1 in /srv/cdnjs/ajax/libs, I get the following:

where e.g. the leftmost bin counts files (or in this case, probably directory entries) with a size 1B <= size < 10B.

Of course, for the inode calculation, only the average size per file is relevant. In this case, this is 14957 bytes/file, so approx 90% of the normal inode/file ratio. If we double the ratio, we should be fine, I think, but for safety (git will probably also create a lot of small files on pull?) we can go for 4kB/inode?

So, shall we just raise the default inode count for labs_lvm::volume or have a hiera setting? Either way, patches are welcome :)

The problem is only relevant for tools-static-*, so I wouldn't change the default for all instances. 4 kByte/inode looks alright to me; we can always recreate the instances in the future if we hit another wall.

ok, so I deleted tools-static-01, I guess that means this can be closed?

The newer hosts have a 60GB partition instead of a 20GB one.

For reference:

• tools-static-01 had a 21G cdnjs drive with 1403520 inodes
• tools-web-static-01 has a 61G drive with 4030464 inodes (1467806 in use).
• tools-web-static-02 is the same as -01 (well, 10 more inodes in use, but who's counting)

As long as we do not use /srv for more than just cdnjs, we're probably fine. On the other hand, chances are we will add some other static stuff there, which likely also consists of many small files, so we'd see the issue return.