Page MenuHomePhabricator

test importing of mailing list configs and archives on staging VM
Closed, ResolvedPublic

Description

after configs and archives have been copied over from sodium,

test importing configs and archives on the new staging host

Details

Related Gerrit Patches:

Related Objects

Event Timeline

Dzahn created this task.Aug 5 2015, 6:12 PM
Dzahn raised the priority of this task from to High.
Dzahn updated the task description. (Show Details)
Dzahn added subscribers: MZMcBride, Dzahn, faidon and 4 others.

try if we can get way with just rsyncing everything, including mbox files and HTML archives, _without_ having to regenerate HTML archives to avoid URL breakage

Dzahn added a comment.Aug 7 2015, 8:58 PM

copied just the config of list "test" from sodium and put it in place on fermium

first there was a mailman bug page but the issues were just file system permissions

(also check the "en" subdirectories and similar, multi-language listinfo pages!)

then we ran withlist -l -r fix_url test -v which fixes the URLs inside the .pck config files and adjusts them to the current server (fermium)

additionally we used mmsitepass to reset the site password, literally _re_set to the same password it was before but it is necessary to be able to login on the imported list using the site pass. the regular list admin pass works without changes

after setting the list to advertised it also showed up on the global listinfo overview

Dzahn added a comment.Aug 7 2015, 10:29 PM

tested import of archives for list test (private) archives. there are 2 directories ./private and public/ , all actual files are in private and the public lists are symlinked into the public dir.

inside ./private there are again 2 directories, <listname> and <listname>.mbox , the former are the HTML files and the latter is the mbox file. Both can simply be rsynced. Did not regenerate anything and did not touch the .mbox file, just permissions have to be right for it to be writable once mail gets delivered.

example
http://fermium.eqiad.wmnet/cgi-bin/mailman/private/test/2012-July/000000.html

Change 231333 had a related patch set uploaded (by Dzahn):
fermium: ferm rule to allow rsync from sodium

https://gerrit.wikimedia.org/r/231333

Change 231333 merged by Dzahn:
fermium: ferm rule to allow rsync from sodium

https://gerrit.wikimedia.org/r/231333

Change 231394 had a related patch set uploaded (by Dzahn):
fermium: adjust rsync import dir, ensure package

https://gerrit.wikimedia.org/r/231394

Change 231394 merged by Dzahn:
fermium: adjust rsync import dir, ensure package

https://gerrit.wikimedia.org/r/231394

Dzahn added a comment.Aug 15 2015, 1:37 AM
#!/bin/bash
# import a mailman list - config and archives
# dzahn@wikimedia.org - 20150814 - T108073

LISTNAME=$1
IMPORT_DIR="/var/lib/mailman/import"
INSTALL_DIR="/var/lib/mailman"
INSTALL_USER="list"

# import a list
importlist () {
 mkdir ${INSTALL_DIR}/lists/${LISTNAME}
 rsync -avp ${IMPORT_DIR}/lists/${LISTNAME}/ ${INSTALL_DIR}/lists/${LISTNAME}/
 chown root:${INSTALL_USER} ${INSTALL_DIR}/lists/${LISTNAME}

 mkdir ${INSTALL_DIR}/archives/private/${LISTNAME}
 chown root:${INSTALL_USER} ${INSTALL_DIR}/archives/private/${LISTNAME}
 rsync --remove-source-files -avp ${IMPORT_DIR}/archives/private/${LISTNAME}/ ${INSTALL_DIR}/archives/private/${LISTNAME}/

 mkdir ${INSTALL_DIR}/archives/private/${LISTNAME}.mbox
 chown root:${INSTALL_USER} ${INSTALL_DIR}/archives/private/${LISTNAME}.mbox
 rsync --remove-source-files -avp ${IMPORT_DIR}/archives/private/${LISTNAME}.mbox/ ${INSTALL_DIR}/archives/private/${LISTNAME}.mbox/

 ${INSTALL_DIR}/bin/withlist -l -r fix_url ${LISTNAME} -v
}

# check if a list is private
isprivate () {

 PRIVATE="unknown"

 if [ ! -d ${IMPORT_DIR}/archives/private/${LISTNAME} ] ; then
    echo "list not found"
    exit 1
 elif [ -h ${IMPORT_DIR}/archives/public/${LISTNAME} ] ; then
    echo "list has a public archive"
    PRIVATE="PUBLIC"
 else
    echo "list is private"
    PRIVATE="PRIVATE"
 fi
}

# usage
if [ -z ${1+x} ]; then
 echo "usage: ./import_list <listname>"
 exit 1
fi

# main
echo "checking list ${LISTNAME}"

isprivate
if [ $PRIVATE == "PUBLIC" ] ; then
    echo -e "importing ${LISTNAME}\n"
    importlist
else
    echo "skipping ${LISTNAME}"
fi

echo -e "--------------------------\n"
#!/bin/bash
# import all lists found in import directory

INSTALL_DIR="/var/lib/mailman"
IMPORT_DIR="/var/lib/mailman/import"

for list in $(find ${IMPORT_DIR}/lists -maxdepth 1) ; do

    LISTNAME=$(basename $list)
    ${INSTALL_DIR}/import_list.sh $LISTNAME
done

echo "done importing. running list_lists\n"
${INSTALL_DIR}/bin/list_lists
Dzahn added a comment.Aug 15 2015, 1:46 AM

one issue with a list that has "locked" in the name , which stopped the import script

Dzahn added a comment.Aug 15 2015, 2:15 AM
==> /var/log/mailman/mischief <==
Aug 15 02:14:58 2015 (3431) Hostile listname: wikitech-announce.disabled.T100503

^ issue when trying to open global listinfo page

Dzahn added a comment.Aug 15 2015, 2:29 AM

"wikiit-l" broke everything, the listinfo page, manual ./list_lists and even a service restart because that also tries ./list_lists -b.

strace helped me find the bad list and after deleting that list directory the rest worked normal again.


now see:

http://fermium.eqiad.wmnet/cgi-bin/mailman/listinfo

all public lists imported, except wikitech-announce.disabled and "wikiit-l"

Dzahn closed this task as Resolved.Aug 15 2015, 2:30 AM
Dzahn claimed this task.

"wikiit-l" broke everything, the listinfo page, manual ./list_lists and even a service restart because that also tries ./list_lists -b.

A re-import shows bad permissions as the cause. wikiit-l was owned by root and 1000 (spamd on sodium; nothing on fermium). Fixing permissions to www-data:list resolved it.

==> /var/log/mailman/mischief <==
Aug 15 02:14:58 2015 (3431) Hostile listname: wikitech-announce.disabled.T100503

^ issue when trying to open global listinfo page

Mailman has list creation standards which are lower case only. Since the T is not lowercase, the list can not be created as is considered 'Hostile' (ironically that has a capital :) ). So rename the list away from a T to a t or delete it.

Clarify: List *names* can be capital. List identifiers (the /listinfo/* part and directories and what Mailman knows the list as) can't.

Dzahn reopened this task as Open.Aug 18 2015, 4:38 PM

Change 232287 had a related patch set uploaded (by Dzahn):
mailman: add helper scripts for importing lists

https://gerrit.wikimedia.org/r/232287

Change 232287 merged by Dzahn:
mailman: add helper scripts for importing lists

https://gerrit.wikimedia.org/r/232287

A re-import shows bad permissions as the cause. wikiit-l was owned by root and 1000 (spamd on sodium; nothing on fermium). Fixing permissions to www-data:list resolved it.

Yes, could not recreate the problem. The permission problem just happened on the second attempt though. First time it was rsync -avp and all lists were copied the same way.

Either way, tried it one more time just now and no problem. Call it a glitch.

Dzahn closed this task as Resolved.Aug 19 2015, 12:11 AM

Now also ran the import test with all private lists. Works. listinfo page still showing fine.