We often have monitoring alerts when servers are about to run out of disk.

One random example being T392834 for mwmaint. This was created in response to T392834#10780146 but is not supposed to be specific to that machine.. it's about all disk space checks.

These alerts only show up on IRC and are often overlooked nowadays. The appropriate owners/subteams are not automatically informed about it.

Then other people jump in to fix it because it's considered an UBN.. or nobody at all and it's only noticed once services actually go down with machines having 0 bytes free.

Once an immediate UBN has been fixed there is usually the need for a follow-up to avoid this from happening again in the future which raises questions for the actual service owners.

A better way approach would be if this alert could also create automatic tickets (like for example RAID checks already do), or would send emails.

Ideally the owner team would be tagged automatically and/or the emails would go to the correct subteam, though this might be more tricky as it would require the monitoring system to know which machine is using which role and lookup the role_contact in the puppet repo.

The lower hanging fruit would be to just switch this disk space check to automatic tickets since we already have that for other checks. Clinic duty or whoever can still tag them with the right team manually.