after fermium is reinstalled:
import all lists with the script we wrote for that
Description
Description
Details
Details
Related Gerrit Patches:
| operations/puppet : production | mailman: remove import scripts |
| operations/puppet : production | mailman: import even unknown lists |
| operations/puppet : production | mailman: also import held messages and qfiles |
| operations/puppet : production | mailman: Don't store bad messages in qfiles |
| operations/puppet : production | mailman: adjust import_list.sh for private lists |
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Resolved | faidon | T84041 Replace all instances of lighttpd with nginx | ||
| Resolved | faidon | T84053 mailman - replace lighttpd | ||
| Open | None | T111653 Encrypt all the things | ||
| Resolved | faidon | T82576 Enable STARTTLS (both inbound and outbound) on lists | ||
| Resolved | JanZerebecki | T55259 Add Forward Secrecy to all HTTPS sites | ||
| Resolved | Dzahn | T90351 Improve SSL of lists.wikimedia.org | ||
| Resolved | Dzahn | T83541 Upgrade Exim to >=4.73 | ||
| Duplicate | None | T97492 Upgrade to Mailman 3.0 | ||
| Resolved | Dzahn | T110141 TTL back up to normal 1H | ||
| Resolved | MZMcBride | T27231 Mailman mailing list archiver truncates if a line begins with "From" | ||
| Resolved | None | T66818 Mitigate strict DMARC policy on the mailing lists | ||
| Resolved | Dzahn | T80945 Get rid of all Ubuntu Lucid (10.04) installs | ||
| Resolved | Dzahn | T82698 shutdown sodium after mailman has migrated to jessie VM | ||
| Resolved | Dzahn | T105756 Mailman Upgrade (Jessie & Mailman 2.x) and migration to a VM | ||
| Invalid | Dzahn | T110131 import all lists with the script we wrote for that |
Event Timeline
Comment Actions
Change 234032 had a related patch set uploaded (by Dzahn):
mailman: adjust import_list.sh for private lists
Comment Actions
Change 234043 had a related patch set uploaded (by Dzahn):
mailman: Don't store bad messages in qfiles
Comment Actions
Change 234138 had a related patch set uploaded (by Dzahn):
mailman: also import held messages and qfiles
Comment Actions
sizes after considerable cleanup (qfiles/bad, shunt, Gigabytes deleted):
17:54 <mutante> 91G archives
17:54 <mutante> 4.4G data
17:54 <mutante> 552M lists
17:54 <mutante> 68M qfiles
Comment Actions
./bin/list_lists says
556 matching mailing lists found on fermium. But the same command says 558 (!) on sodium. Figure out the diff!
-->
Licom-l - Wikimedia licensing update committee (CLOSED)
WLM-CN - Wiki Loves Monuments In China
Comment Actions
it probably has to do with the way these have been disabled in the past.
i see:
wlm-cn.disabled.rt8567 and licom-l.disabled.rt7307 on sodium in ./lists/. These have been rsynced but not imported. "list not found" by import script. Because of the dots in the names?
Comment Actions
Importing fix_url...
Running fix_url.fix_url()...
Loading list foundation-l (locked)
Unknown list: foundation-l
Traceback (most recent call last):
File "/var/lib/mailman/bin/withlist", line 299, in <module>
main()
File "/var/lib/mailman/bin/withlist", line 277, in main
r = do_list(listname, args, func)
File "/var/lib/mailman/bin/withlist", line 202, in do_list
return func(m, *args)
File "/usr/lib/mailman/bin/fix_url.py", line 73, in fix_url
if not mlist.Locked():
AttributeError: 'NoneType' object has no attribute 'Locked'
--------------------------Comment Actions
Lists to check for import issues (because there are still files left that should have been deleted if it was succesful).
advocacy_advisors
analytics.backup
boardexec
checkuser-l.backup
chip-l
clevel
education-coop
foundation-news-l
fundraising.backup
fundraising-de
gendergap.backup.20110209
hiphop
juribak
langcom-observers
licom-l
mk-edu
moderators-nl.save
officeit
private-l
research-team
tools-wmt-staff
unblock-en-l.backup
wikibooksde-l
wikidata-l.backup
wikidata.old.backup
wikide-l.backup
wikimedia-commits
wikimediacz-oldprivate
wikimedia-de-by-ltp
wikimedia-in
wikimedia-l.old
wikimedia-ve
wikimedia-ve-afiliados
wikimedia-ve-entusiastas
wikimedical-l
wikipedia-ensino
wiki-research-l.backup
wikitech-announce
wikitech-announce.disabled.t100503
wlm-cn
wmf-care-bears
wmfresearch
wmfsocial
xmldatadumps-admin-l
Comment Actions
Glancing over the above list and recalling memories related to them (but not tickets right now; perhaps will track these tomorrow if needed), they're not real lists. Some are either deleted (tools-wmt-staff as I requested it) or disabled in a bad way (Wikitech-announce as an example, renamed twice by directory).
Comment Actions
We should just blindly rsync it all so it's exactly like before and if in doubt we can deal with it any time on the new server. That means changing the script to not even attempt checking whether a list exists from MM's point of view.
Comment Actions
+1 I don't see why is a special purpose script even needed.
Also, please check that non-ascii characters are ok in the new mailman. We had charset problems there when upgrading WM-ES lists from wheezy to jessie, to the point that some lists had special chars in some fields that made python raised an encoding exception (other fields like the list footer just happily send the garbage).
Comment Actions
It started as "for the very first tests i just want to export public data and not mess with the private things before i;m more sure it even works".
Also, please check that non-ascii characters are ok in the new mailman. We had charset problems there when upgrading WM-ES lists from wheezy to jessie, to the point that some lists had special chars in some fields that made python raised an encoding exception (other fields like the list footer just happily send the garbage).
We even had a puppet failure "failed: invalid byte sequence in UTF-8" because of one charset issue in one template, the French listinfo template.
It was: listinfo.html: HTML document, ISO-8859 text
and i ran iconv -f ISO-8859-1 -t UTF8
and then it was: listinfo.html: HTML document, UTF-8 Unicode text
https://gerrit.wikimedia.org/r/#/c/234589/
and that fixed the puppet run on jessie.
But that's the only file i touched encoding-wise,i left all the other language templates alone because they did not cause puppet issues.
Even though they look like they are in different formats. --> see P1944
At first i planned to convert them all to UTF-8, like here: https://gerrit.wikimedia.org/r/#/c/234565/ but that looked like it broke more than it did good, for example see Korean.
I then decided to leave them untouched except the one that caused actual issues. So, French fixed, all others as on sodium, for now.
Comment Actions
Change 237011 had a related patch set uploaded (by Dzahn):
mailman: import even unknown lists
Comment Actions
Change 237315 had a related patch set uploaded (by Dzahn):
mailman: remove import scripts
Comment Actions
we won't use that script anymore. instead we adjusted the rsyncd config and the script running on the source side to directly sync into /var/lib/mailman in the right places