Note that due to T198679: Investigate switching internal mailing list over to GSuite group we are looking for a quick fix, nothing more.

The issue seems to be qith e-mails not being delivered to mailman (after 24ish hours there is a "421 Unexpected failure, please try later"). Verified that e-mails are not found in the mailman archives.

The mailman interface is reachable, but the admin interface throwa a bug (stacktrace need to be accessed theough server log).

There have been recent changes to some of our other servers (T232567) and spf records (T232726) which, while unlikely, may have a bearing on the issue.

Looks like the disc is full, largely due to logfiles

For future reference:

• Tried to edit my crontab (to kill a daily failing cron job) which warned me we were out of disc space.
• Confirmed this by logging into the glesys dashboard.
• From the dashboard i logged into a root console and ran du -a / | sort -n -r | head -n 20 to give me the largest folders. The main culprits were:
  • /var/log/apache2/ with /var/log/apache2/error.log leading the race.
    • For the various old disabled sites:
    • I nuked the logs
    • cleaned them out of /etc/apache2/sites-enabled/
    • cleaned them out of /etc/apache2/sites-available/
    • ensured the subdomain was removed from loopia
    • deleted the data folder from /home/www/deleteme/

Exceptions:

• wlm.wikimedia.se (left everything other than old logs) because it does more things under the hood
• piwik.wikimedia.se (left everything other than old logs) although the domain points to FS-data I figured this can be left until T206773: Reenable Matomo [previously Piwik] is done, in case there is anything here we want
• 002-styrelse.wikimedia.se (left everything) I have a sneaky feeling this gets used for mailman

The sad thing is that we are still using 9/10GB. But at least I can now get into the admin interface so lets see if the e-mails now arrive.

E-mails are working again