Page MenuHomePhabricator

MariaDB disk space check should exclude ramfs in addition to tmpfs filesystems
Closed, ResolvedPublic

Description

Today, a sort of race condition happened, where systemd mounted temporarily:

ramfs on /run/credentials/systemd-tmpfiles-clean.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)

which led to a p@ging alert on es1036: with /run/credentials/systemd-tmpfiles-clean.service is not accessible: Permission denied

image.png (113×1 px, 74 KB)

I have never seen this false positive happen on databases, but it seems that the right fix is to ignore ramfs types from the check, as we are usually only interested on ext4, xfs and maybe other "real" filesystems mounted. In particular, even df ignores by default all "dummy, duplicate, or inaccessible file systems, which are omitted by default. Dummy file systems are typically special purpose pseudo file systems such as ‘/proc’, with no associated storage."

I don't know why this particular instance failed, maybe because it was temporary there is a non-atomic "listing -> umount -> check" that leads to a race condition? But that wouldn't make sense, it wouldn't have failed 3 times in a row. Maybe the cleaning of temporary files lasted more time than usual?

Event Timeline

jcrespo triaged this task as Unbreak Now! priority.Jul 1 2025, 6:15 AM

The easy fix probably should be to add ramfs to excludes of modules/mariadb/manifests/monitor_disk.pp but that is a delicate change, as this p@ges on every host- needs proper testing.

To exclude ramfs and tmpfs filesystems when checking MariaDB disk space, you can use the df command with specific options. Here's how to approach this:

Understanding ramfs and tmpfs
ramfs and tmpfs are temporary file systems that store data in RAM. Unlike regular file systems, they don't occupy fixed disk space, making them unsuitable for MariaDB data storage due to data loss upon system reboot ¹.

Checking Disk Space
To check disk space excluding ramfs and tmpfs, you can use the df command with the -x option, which excludes specific filesystem types. Here's an example:
df -x tmpfs -x ramfs

This command will display disk usage statistics for all mounted filesystems except tmpfs and ramfs.

Optimizing MariaDB Disk Space
To optimize MariaDB disk space, consider the following steps ² ³:

  • *Optimize InnoDB Log Files*: Adjust the innodb_log_file_size variable to control the size of log files.
  • *Run OPTIMIZE TABLE*: Execute OPTIMIZE TABLE commands to reorganize and optimize storage space used by the database.
  • *Check for Disk Space Allocation*: Use queries to retrieve disk space allocated by datafiles but not used.

Additional Tips

  • Regularly monitor disk space usage to prevent running out of space.
  • Consider automating optimization using Cron Jobs.
  • Ensure proper configuration of MariaDB to prevent excessive disk space usage ².

Change #1166377 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Exclude tmpfs and ramfs from paging disk monitor alerts

https://gerrit.wikimedia.org/r/1166377

Marostegui subscribed.

Thanks Jaime - I will merge on Monday.

Change #1166377 merged by Marostegui:

[operations/puppet@production] mariadb: Exclude tmpfs and ramfs from paging disk monitor alerts

https://gerrit.wikimedia.org/r/1166377

After merging Jaime's patch I think this is done