migration plan for mailman

objectives:

• move away from server sodium (lucid) to server fermium (jessie) to get rid of the last lucid box in all of WMF
• upgrade mailman from 2.1.13 to 2.1.18

tracking ticket: https://phabricator.wikimedia.org/T105756

Problems a migration will have:

• sodium and fermium would both be acting as mailman servers for lists.wikimedia.org -- solved by stopping mailman and holding exim

• small window of latency for updates/risk of losing changes or actions if performed on sodium while we call fermium 'active'

• emails lost by queued exim in migration -- move queued emails to fermium and hope exim can pick them up there

Tickets that will be solved:
T80945 (get rid of all lucid installs)
T80962 (upgrade mailman version)
T82698 (shutdown sodium)
T66818 (strict DMARC policy) - mailman settings mostly, solved in 2.1.16
T27231 (Mailman mailing list archiver truncates if a line begins with "From") solved by new version, right? fixed in 2.1.16
T90351 (improve SSL config), solved by Apache 2.4 upgrade, already checked
T83967 (cleanup held messages)
T109838 (cleanup held messages 2)
T104980 (ferm rules, they exist but have not been applied on prod., existing manual rules on sodium don't have to be copied)

Tickets that will be unblocked but not solved yet (on purpose, one step at a time):

T82576, T109239 (enabling STARTTLS)

Since I cant edit comments, 14 is done and depends on 15. Since nothing else depends on fermium existence as is and other work is blocked, I'm going to give Alex an okay go move fermium from private to public spaces.

Added the table to the description.

items from meeting:

"double check for mailman cronjobs"
"how long does rsync take" (for the final run, both)
"where does mailman store listinfo info"
"also tell ops list"
"stop exim, copy spool directories"
"exim commands / cheat sheet: show mail queue, send individual mail from queue, then multiple"

qfiles; this can be handled two ways. We could stop mailman with exim and rsync them or we can hold exim and let mailman run for a defined period of time (10 minutes to be safe) before we stop it. This would mean we don't actually have to rsync the qfiles at all and any held mail like pending and so on would also be cleaned up which makes an rsync easier, cleaner and reduces chance of duplicating out-going emails.

"how long does rsync take" -> an hour or a little bit more, depending on how long we wait between rsync runs. actually tested this though, lowered time a bit by deleting a bunch of crap from qfiles/bad and shunt files older than 7 days

"double check cronjobs" -> T110382

"where does store listinfo info" -> in each lists config.pck and we run "withlist" to update it after import but this is only necessary when URL actually changes

"also tell ops" -> added on T110133
"stop exim, copy spool directories" -> T110440, T110136
"exim commands cheat sheet" -> T110441