after configs and archives have been copied over from sodium,
test importing configs and archives on the new staging host
Description
Description
Details
Details
Event Timeline
Comment Actions
try if we can get way with just rsyncing everything, including mbox files and HTML archives, _without_ having to regenerate HTML archives to avoid URL breakage
Comment Actions
copied just the config of list "test" from sodium and put it in place on fermium
first there was a mailman bug page but the issues were just file system permissions
(also check the "en" subdirectories and similar, multi-language listinfo pages!)
then we ran withlist -l -r fix_url test -v which fixes the URLs inside the .pck config files and adjusts them to the current server (fermium)
additionally we used mmsitepass to reset the site password, literally _re_set to the same password it was before but it is necessary to be able to login on the imported list using the site pass. the regular list admin pass works without changes
after setting the list to advertised it also showed up on the global listinfo overview
Comment Actions
tested import of archives for list test (private) archives. there are 2 directories ./private and public/ , all actual files are in private and the public lists are symlinked into the public dir.
inside ./private there are again 2 directories, <listname> and <listname>.mbox , the former are the HTML files and the latter is the mbox file. Both can simply be rsynced. Did not regenerate anything and did not touch the .mbox file, just permissions have to be right for it to be writable once mail gets delivered.
example
http://fermium.eqiad.wmnet/cgi-bin/mailman/private/test/2012-July/000000.html
Comment Actions
Change 231333 had a related patch set uploaded (by Dzahn):
fermium: ferm rule to allow rsync from sodium
Comment Actions
Change 231394 had a related patch set uploaded (by Dzahn):
fermium: adjust rsync import dir, ensure package
Comment Actions
#!/bin/bash # import a mailman list - config and archives # dzahn@wikimedia.org - 20150814 - T108073 LISTNAME=$1 IMPORT_DIR="/var/lib/mailman/import" INSTALL_DIR="/var/lib/mailman" INSTALL_USER="list" # import a list importlist () { mkdir ${INSTALL_DIR}/lists/${LISTNAME} rsync -avp ${IMPORT_DIR}/lists/${LISTNAME}/ ${INSTALL_DIR}/lists/${LISTNAME}/ chown root:${INSTALL_USER} ${INSTALL_DIR}/lists/${LISTNAME} mkdir ${INSTALL_DIR}/archives/private/${LISTNAME} chown root:${INSTALL_USER} ${INSTALL_DIR}/archives/private/${LISTNAME} rsync --remove-source-files -avp ${IMPORT_DIR}/archives/private/${LISTNAME}/ ${INSTALL_DIR}/archives/private/${LISTNAME}/ mkdir ${INSTALL_DIR}/archives/private/${LISTNAME}.mbox chown root:${INSTALL_USER} ${INSTALL_DIR}/archives/private/${LISTNAME}.mbox rsync --remove-source-files -avp ${IMPORT_DIR}/archives/private/${LISTNAME}.mbox/ ${INSTALL_DIR}/archives/private/${LISTNAME}.mbox/ ${INSTALL_DIR}/bin/withlist -l -r fix_url ${LISTNAME} -v } # check if a list is private isprivate () { PRIVATE="unknown" if [ ! -d ${IMPORT_DIR}/archives/private/${LISTNAME} ] ; then echo "list not found" exit 1 elif [ -h ${IMPORT_DIR}/archives/public/${LISTNAME} ] ; then echo "list has a public archive" PRIVATE="PUBLIC" else echo "list is private" PRIVATE="PRIVATE" fi } # usage if [ -z ${1+x} ]; then echo "usage: ./import_list <listname>" exit 1 fi # main echo "checking list ${LISTNAME}" isprivate if [ $PRIVATE == "PUBLIC" ] ; then echo -e "importing ${LISTNAME}\n" importlist else echo "skipping ${LISTNAME}" fi echo -e "--------------------------\n"
#!/bin/bash # import all lists found in import directory INSTALL_DIR="/var/lib/mailman" IMPORT_DIR="/var/lib/mailman/import" for list in $(find ${IMPORT_DIR}/lists -maxdepth 1) ; do LISTNAME=$(basename $list) ${INSTALL_DIR}/import_list.sh $LISTNAME done echo "done importing. running list_lists\n" ${INSTALL_DIR}/bin/list_lists
Comment Actions
one issue with a list that has "locked" in the name , which stopped the import script
Comment Actions
==> /var/log/mailman/mischief <== Aug 15 02:14:58 2015 (3431) Hostile listname: wikitech-announce.disabled.T100503
^ issue when trying to open global listinfo page
Comment Actions
"wikiit-l" broke everything, the listinfo page, manual ./list_lists and even a service restart because that also tries ./list_lists -b.
strace helped me find the bad list and after deleting that list directory the rest worked normal again.
now see:
http://fermium.eqiad.wmnet/cgi-bin/mailman/listinfo
all public lists imported, except wikitech-announce.disabled and "wikiit-l"
Comment Actions
A re-import shows bad permissions as the cause. wikiit-l was owned by root and 1000 (spamd on sodium; nothing on fermium). Fixing permissions to www-data:list resolved it.
Comment Actions
Mailman has list creation standards which are lower case only. Since the T is not lowercase, the list can not be created as is considered 'Hostile' (ironically that has a capital :) ). So rename the list away from a T to a t or delete it.
Comment Actions
Clarify: List *names* can be capital. List identifiers (the /listinfo/* part and directories and what Mailman knows the list as) can't.
Comment Actions
Change 232287 had a related patch set uploaded (by Dzahn):
mailman: add helper scripts for importing lists
Comment Actions
Yes, could not recreate the problem. The permission problem just happened on the second attempt though. First time it was rsync -avp and all lists were copied the same way.
Either way, tried it one more time just now and no problem. Call it a glitch.
Comment Actions
Now also ran the import test with all private lists. Works. listinfo page still showing fine.