/srv on tools-static-01 running out of inodes
Closed, ResolvedPublic


Tools-static-01 is unable to git-pull in /srv/cdnjs because the volume is out of inodes. It looks like a legit problem, like cdnjs just has that many files.

Andrew created this task.Aug 1 2015, 2:44 PM
Andrew updated the task description. (Show Details)
Andrew raised the priority of this task from to Needs Triage.
Andrew added projects: Cloud-Services, Toolforge.
Andrew added subscribers: Andrew, yuvipanda, coren.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2015, 2:44 PM
scfc added a subscriber: scfc.Aug 1 2015, 5:12 PM

The underlying problem is in modules/toollabs/manifests/static.pp:

labs_lvm::volume { 'cdnjs-disk':
    mountat => '/srv',
    size    => '100%FREE'

This uses the standard space/inode ratio of mkfs.ext4. It should pass a mkfs_opt parameter with the needed options. Then creating new instances should fix the issue (inodes cannot be added to an existing file system).

scfc moved this task from Triage to Backlog on the Toolforge board.Aug 1 2015, 5:12 PM
valhallasw triaged this task as High priority.Aug 2 2015, 1:14 PM
valhallasw added a subscriber: valhallasw.

If I understand correctly, this is currently breaking cdnjs updates (or even just git cloneing it? not sure)? If not, please reduce the priority to 'normal'.

Andrew added a comment.Aug 3 2015, 4:22 AM

Is is breaking updates, yes.

You say that the problem is 'This uses the standard space/inode ratio'. Is the standard ratio 'barely any inodes'? Or is cdnjs absurdly file-heavy?

So they are currently being served out of tools-web-static-xx or something - which have bigger disks :) but yes cdnjs is crazily inode heavy. Is this problem in the old nodes which I might not have deleted or the new ones?

Sitic added a subscriber: Sitic.Aug 3 2015, 12:31 PM
scfc added a comment.Aug 3 2015, 2:02 PM

(The standard space/inode ratio for ext4 is about 16384 bytes/inode.)

It's somewhat inode heavy, but not even that crazily. Based on du -aSb | cut -f1 in /srv/cdnjs/ajax/libs, I get the following:

where e.g. the leftmost bin counts files (or in this case, probably directory entries) with a size 1B <= size < 10B.

Of course, for the inode calculation, only the average size per file is relevant. In this case, this is 14957 bytes/file, so approx 90% of the normal inode/file ratio. If we double the ratio, we should be fine, I think, but for safety (git will probably also create a lot of small files on pull?) we can go for 4kB/inode?

Andrew added a comment.Aug 3 2015, 2:54 PM

So, shall we just raise the default inode count for labs_lvm::volume or have a hiera setting? Either way, patches are welcome :)

scfc added a comment.Aug 3 2015, 3:07 PM

The problem is only relevant for tools-static-*, so I wouldn't change the default for all instances. 4 kByte/inode looks alright to me; we can always recreate the instances in the future if we hit another wall.

Andrew added a comment.Aug 3 2015, 7:14 PM

ok, so I deleted tools-static-01, I guess that means this can be closed?

The newer hosts have a 60GB partition instead of a 20GB one.

For reference:

  • tools-static-01 had a 21G cdnjs drive with 1403520 inodes
  • tools-web-static-01 has a 61G drive with 4030464 inodes (1467806 in use).
  • tools-web-static-02 is the same as -01 (well, 10 more inodes in use, but who's counting)

As long as we do not use /srv for more than just cdnjs, we're probably fine. On the other hand, chances are we will add some other static stuff there, which likely also consists of many small files, so we'd see the issue return.

Andrew closed this task as Resolved.Aug 6 2015, 4:12 PM
Andrew claimed this task.
Andrew moved this task from To Do to Done on the Labs-Sprint-108 board.