Change Details

XFS filesystems running on 4.9 kernels showed negative free space, the problem appeared to be a filesystem problem as [[ https://github.com/monitoring-plugins/monitoring-plugins/issues/1544 | both the nagios check and df showed the negative free space ]]. - Original ticket, https://phabricator.wikimedia.org/T199198 - Workaround by disabling free inode btree at mkfs time, https://gerrit.wikimedia.org/r/c/operations/puppet/+/472415 - Icinga check: https://phabricator.wikimedia.org/T199436 Most of the reports on the internet about negative free space on XFS seem to occur around the same time frame. Then a commit was made to 4.18 which fixes, [[ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=aafe12cee0b132824f5187987f8a3fb704b9f685 | xfs: don't trip over negative free space in xfs_reserve_blocks ]]. I wasn't able to find any reports after this commit, so I think there is a decent chance this commit fixed it or another one did around the same time. Since we still have the workaround in place and since all buster and greater servers are running kernels newer than 4.18 my vote is to leave this check in icinga, util the swift nodes are upgraded off of stretch.

XFS filesystems running on 4.9 kernels showed negative free space, the problem appeared to be a filesystem problem as [[ https://github.com/monitoring-plugins/monitoring-plugins/issues/1544 | both the nagios check and df showed the negative free space ]]. - Original ticket {T199198} - Workaround by disabling free inode btree at mkfs time, https://gerrit.wikimedia.org/r/c/operations/puppet/+/472415 - Icinga check: {T199436} Most of the reports on the internet about negative free space on XFS seem to occur around the same time frame. Then a commit was made to 4.18 which fixes, [[ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=aafe12cee0b132824f5187987f8a3fb704b9f685 | xfs: don't trip over negative free space in xfs_reserve_blocks ]]. I wasn't able to find any reports after this commit, so I think there is a decent chance this commit fixed it or another one did around the same time. Since we still have the workaround in place and since all buster and greater servers are running kernels newer than 4.18 my vote is to leave this check in icinga, util the swift nodes are upgraded off of stretch.

XFS filesystems running on 4.9 kernels showed negative free space, the problem appeared to be a filesystem problem as [[ https://github.com/monitoring-plugins/monitoring-plugins/issues/1544 | both the nagios check and df showed the negative free space ]]. - Original ticket, https://phabricator.wikimedia.org/ {T199198} - Workaround by disabling free inode btree at mkfs time, https://gerrit.wikimedia.org/r/c/operations/puppet/+/472415 - Icinga check: https://phabricator.wikimedia.org/{T199436} Most of the reports on the internet about negative free space on XFS seem to occur around the same time frame. Then a commit was made to 4.18 which fixes, [[ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=aafe12cee0b132824f5187987f8a3fb704b9f685 | xfs: don't trip over negative free space in xfs_reserve_blocks ]]. I wasn't able to find any reports after this commit, so I think there is a decent chance this commit fixed it or another one did around the same time. Since we still have the workaround in place and since all buster and greater servers are running kernels newer than 4.18 my vote is to leave this check in icinga, util the swift nodes are upgraded off of stretch.