Page MenuHomePhabricator

Internal mailing lists are unresponsive
Closed, ResolvedPublic

Description

Our three internal mailinglists have been unresponsive since the 2019-09-24 at least.

Investigate why and bring them online again.

Event Timeline

Lokal_Profil renamed this task from Internal mailing lists are unreaponsive to Internal mailing lists are unresponsive.Sep 25 2019, 1:40 PM
Lokal_Profil created this task.
Lokal_Profil updated the task description. (Show Details)

Note that due to T198679: Investigate switching internal mailing list over to GSuite group we are looking for a quick fix, nothing more.

The issue seems to be qith e-mails not being delivered to mailman (after 24ish hours there is a "421 Unexpected failure, please try later"). Verified that e-mails are not found in the mailman archives.

The mailman interface is reachable, but the admin interface throwa a bug (stacktrace need to be accessed theough server log).

There have been recent changes to some of our other servers (T232567) and spf records (T232726) which, while unlikely, may have a bearing on the issue.

Looks like the disc is full, largely due to logfiles

For future reference:

  • Tried to edit my crontab (to kill a daily failing cron job) which warned me we were out of disc space.
  • Confirmed this by logging into the glesys dashboard.
  • From the dashboard i logged into a root console and ran du -a / | sort -n -r | head -n 20 to give me the largest folders. The main culprits were:
    • /var/log/apache2/ with /var/log/apache2/error.log leading the race.
      • For the various old disabled sites:
      • I nuked the logs
      • cleaned them out of /etc/apache2/sites-enabled/
      • cleaned them out of /etc/apache2/sites-available/
      • ensured the subdomain was removed from loopia
      • deleted the data folder from /home/www/deleteme/

Exceptions:

  • wlm.wikimedia.se (left everything other than old logs) because it does more things under the hood
  • piwik.wikimedia.se (left everything other than old logs) although the domain points to FS-data I figured this can be left until T206773: Reenable Matomo [previously Piwik] is done, in case there is anything here we want
  • 002-styrelse.wikimedia.se (left everything) I have a sneaky feeling this gets used for mailman

The sad thing is that we are still using 9/10GB. But at least I can now get into the admin interface so lets see if the e-mails now arrive.

Lokal_Profil moved this task from Backlog to Done on the WMSE (IT) board.
Lokal_Profil moved this task from 📆 This week to ☑️ Done on the User-LokalProfil board.