Page MenuHomePhabricator
Create Task
Maniphest T110131

import all lists with the script we wrote for that
Closed, InvalidPublic
Actions

Description

after fermium is reinstalled:

import all lists with the script we wrote for that

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -122mailman: remove import scripts
operations/puppetproduction+12 -11mailman: import even unknown lists
operations/puppetproduction+7 -1mailman: also import held messages and qfiles
operations/puppetproduction+3 -0mailman: Don't store bad messages in qfiles
operations/puppetproduction+4 -1mailman: adjust import_list.sh for private lists
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Aug 24 2015, 11:27 PM
Dzahn claimed this task.
Dzahn raised the priority of this task from to High.
Dzahn updated the task description. (Show Details)
Dzahn added projects: acl*sre-team, Wikimedia-Mailing-lists.
Dzahn added subscribers: ori, MZMcBride, Dzahn and 5 others.
Dzahn mentioned this in T109467: write migration plan for mailman.Aug 24 2015, 11:59 PM
• JohnLewis moved this task from Backlog to Jessie/Upgrade (2.x) on the Wikimedia-Mailing-lists board.Aug 25 2015, 12:23 PM
gerritbot added a subscriber: gerritbot.Aug 26 2015, 6:30 PM

Change 234032 had a related patch set uploaded (by Dzahn):
mailman: adjust import_list.sh for private lists

https://gerrit.wikimedia.org/r/234032

gerritbot added a project: Patch-For-Review.Aug 26 2015, 6:30 PM
gerritbot added a comment.Aug 26 2015, 7:12 PM

Change 234032 merged by Dzahn:
mailman: adjust import_list.sh for private lists

https://gerrit.wikimedia.org/r/234032

Dzahn mentioned this in rOPUPb3ad262bc6ee: mailman: adjust import_list.sh for private lists.Aug 26 2015, 7:12 PM
gerritbot added a comment.Aug 26 2015, 7:12 PM

Change 234043 had a related patch set uploaded (by Dzahn):
mailman: Don't store bad messages in qfiles

https://gerrit.wikimedia.org/r/234043

gerritbot added a comment.Aug 26 2015, 7:19 PM

Change 234043 merged by Dzahn:
mailman: Don't store bad messages in qfiles

https://gerrit.wikimedia.org/r/234043

Dzahn mentioned this in rOPUP4335126b74a5: mailman: Don't store bad messages in qfiles.Aug 26 2015, 7:19 PM
gerritbot added a comment.Aug 26 2015, 9:04 PM

Change 234138 had a related patch set uploaded (by Dzahn):
mailman: also import held messages and qfiles

https://gerrit.wikimedia.org/r/234138

Dzahn added a comment.Aug 27 2015, 12:54 AM

sizes after considerable cleanup (qfiles/bad, shunt, Gigabytes deleted):

17:54 <mutante> 91G archives
17:54 <mutante> 4.4G data
17:54 <mutante> 552M lists
17:54 <mutante> 68M qfiles

gerritbot added a comment.Aug 27 2015, 12:57 AM

Change 234138 merged by Dzahn:
mailman: also import held messages and qfiles

https://gerrit.wikimedia.org/r/234138

Dzahn mentioned this in rOPUP1c22d5939538: mailman: also import held messages and qfiles.Aug 27 2015, 12:57 AM
Dzahn added a comment.Aug 27 2015, 4:04 AM

a full import with ./import_all_lists.sh took: real 115m26.589s

Dzahn added a comment.EditedAug 27 2015, 4:19 AM

./bin/list_lists says

556 matching mailing lists found on fermium. But the same command says 558 (!) on sodium. Figure out the diff!

-->

Licom-l - Wikimedia licensing update committee (CLOSED)
WLM-CN - Wiki Loves Monuments In China

Dzahn added a comment.Aug 27 2015, 4:42 AM

it probably has to do with the way these have been disabled in the past.

i see:

wlm-cn.disabled.rt8567 and licom-l.disabled.rt7307 on sodium in ./lists/. These have been rsynced but not imported. "list not found" by import script. Because of the dots in the names?

Dzahn added a comment.Aug 27 2015, 4:49 AM
Importing fix_url...
Running fix_url.fix_url()...
Loading list foundation-l (locked)
Unknown list: foundation-l
Traceback (most recent call last):
  File "/var/lib/mailman/bin/withlist", line 299, in <module>
    main()
  File "/var/lib/mailman/bin/withlist", line 277, in main
    r = do_list(listname, args, func)
  File "/var/lib/mailman/bin/withlist", line 202, in do_list
    return func(m, *args)
  File "/usr/lib/mailman/bin/fix_url.py", line 73, in fix_url
    if not mlist.Locked():
AttributeError: 'NoneType' object has no attribute 'Locked'
--------------------------
Dzahn added a comment.Aug 27 2015, 4:55 AM

Lists to check for import issues (because there are still files left that should have been deleted if it was succesful).

advocacy_advisors
analytics.backup
boardexec
checkuser-l.backup
chip-l
clevel
education-coop
foundation-news-l
fundraising.backup
fundraising-de
gendergap.backup.20110209
hiphop
juribak
langcom-observers
licom-l
mk-edu
moderators-nl.save
officeit
private-l
research-team
tools-wmt-staff
unblock-en-l.backup
wikibooksde-l
wikidata-l.backup
wikidata.old.backup
wikide-l.backup
wikimedia-commits
wikimediacz-oldprivate
wikimedia-de-by-ltp
wikimedia-in
wikimedia-l.old
wikimedia-ve
wikimedia-ve-afiliados
wikimedia-ve-entusiastas
wikimedical-l
wikipedia-ensino
wiki-research-l.backup
wikitech-announce
wikitech-announce.disabled.t100503
wlm-cn
wmf-care-bears
wmfresearch
wmfsocial
xmldatadumps-admin-l

• JohnLewis added a comment.Aug 27 2015, 9:56 PM

Glancing over the above list and recalling memories related to them (but not tickets right now; perhaps will track these tomorrow if needed), they're not real lists. Some are either deleted (tools-wmt-staff as I requested it) or disabled in a bad way (Wikitech-announce as an example, renamed twice by directory).

Dzahn added a comment.Aug 31 2015, 10:16 PM

We should just blindly rsync it all so it's exactly like before and if in doubt we can deal with it any time on the new server. That means changing the script to not even attempt checking whether a list exists from MM's point of view.

Dzahn mentioned this in T105756: Mailman Upgrade (Jessie & Mailman 2.x) and migration to a VM.Sep 1 2015, 8:44 PM
Platonides added a subscriber: Platonides.Sep 2 2015, 9:28 PM
In T110131#1590865, @Dzahn wrote:

We should just blindly rsync it all so it's exactly like before and if in doubt we can deal with it any time on the new server. That means changing the script to not even attempt checking whether a list exists from MM's point of view.

+1 I don't see why is a special purpose script even needed.

Also, please check that non-ascii characters are ok in the new mailman. We had charset problems there when upgrading WM-ES lists from wheezy to jessie, to the point that some lists had special chars in some fields that made python raised an encoding exception (other fields like the list footer just happily send the garbage).

Dzahn added a comment.Sep 2 2015, 10:07 PM
In T110131#1599443, @Platonides wrote:
In T110131#1590865, @Dzahn wrote:

We should just blindly rsync it all so it's exactly like before and if in doubt we can deal with it any time on the new server. That means changing the script to not even attempt checking whether a list exists from MM's point of view.

+1 I don't see why is a special purpose script even needed.

It started as "for the very first tests i just want to export public data and not mess with the private things before i;m more sure it even works".

Also, please check that non-ascii characters are ok in the new mailman. We had charset problems there when upgrading WM-ES lists from wheezy to jessie, to the point that some lists had special chars in some fields that made python raised an encoding exception (other fields like the list footer just happily send the garbage).

We even had a puppet failure "failed: invalid byte sequence in UTF-8" because of one charset issue in one template, the French listinfo template.

It was: listinfo.html: HTML document, ISO-8859 text
and i ran iconv -f ISO-8859-1 -t UTF8
and then it was: listinfo.html: HTML document, UTF-8 Unicode text
https://gerrit.wikimedia.org/r/#/c/234589/

and that fixed the puppet run on jessie.

But that's the only file i touched encoding-wise,i left all the other language templates alone because they did not cause puppet issues.

Even though they look like they are in different formats. --> see P1944

At first i planned to convert them all to UTF-8, like here: https://gerrit.wikimedia.org/r/#/c/234565/ but that looked like it broke more than it did good, for example see Korean.

I then decided to leave them untouched except the one that caused actual issues. So, French fixed, all others as on sodium, for now.

Platonides mentioned this in T39817: lists.wikimedia.org encoding issues in descriptions.Sep 3 2015, 7:52 PM
gerritbot added a comment.Sep 9 2015, 1:24 AM

Change 237011 had a related patch set uploaded (by Dzahn):
mailman: import even unknown lists

https://gerrit.wikimedia.org/r/237011

gerritbot added a comment.Sep 9 2015, 1:27 AM

Change 237011 merged by Dzahn:
mailman: import even unknown lists

https://gerrit.wikimedia.org/r/237011

Dzahn mentioned this in rOPUP6ae2694fd57b: mailman: import even unknown lists.Sep 9 2015, 1:28 AM
gerritbot added a comment.Sep 10 2015, 1:29 AM

Change 237315 had a related patch set uploaded (by Dzahn):
mailman: remove import scripts

https://gerrit.wikimedia.org/r/237315

gerritbot added a comment.Sep 10 2015, 2:03 AM

Change 237315 merged by Dzahn:
mailman: remove import scripts

https://gerrit.wikimedia.org/r/237315

Dzahn mentioned this in rOPUP0d833f96ef06: mailman: remove import scripts.Sep 10 2015, 2:03 AM
Dzahn added a comment.Sep 10 2015, 2:10 AM

we won't use that script anymore. instead we adjusted the rsyncd config and the script running on the source side to directly sync into /var/lib/mailman in the right places

Dzahn closed this task as Invalid.Sep 10 2015, 2:10 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL