Page MenuHomePhabricator

Migrate Wikimini into WMCH infrastructure
Closed, ResolvedPublic

Description

Wikimedia CH is planning to help Wikimini (https://wikimini.org/) in their migration to WMCH infrastructure.

Plan

  • initialize the needed WMCH environment to welcome the new Wikimini's MediaWiki farm (done on server wmch-members2)
  • work on the "Wikimini.org→WMCH" import script (and reduce import time - done from ~4 hours down to ~45 minutes)
  • do a first temporary import
  • put online the new entry point https://wikimini.wikimedia.ch/ - https://sv.wikimini.wikimedia.ch/ ecc.
  • first test period and bug fixes
  • alert https://wikimini.org users about final migration
  • set https://wikimini.org in read-only
  • execute the "Wikimini.org→WMCH" sync again
  • update the DNS records of Wikimini.org
  • set wikimini in write-mode again
  • keep old servers online for a couple of days
  • shutdown old servers
  • dismiss old servers

When

Analyzing the current usage with Matmo, to have a minimal impact, the final migration can be fixed in a Thursday or a Friday, 02:30-04:30 PM CEST.

https://commons.wikimedia.org/wiki/File:Matomo_Visits_per_local_time_-_Wikimini.org_-_February_2021.png

https://commons.wikimedia.org/wiki/File:Matomo_Visits_per_week_day_-_Wikimini.org_-_February_2021.png

Intervention report

Currently it seems they have 8 wikis (ar, en, es, fr, it, lab, stock, sv and a test wiki):

$ mysql
$ SHOW DATABASES;
+---------------------+
| Database            |
+---------------------+
| information_schema  |
| mysql               |
| performance_schema  |
| phpmyadmin          |
| wikimini_arwiki     |
| wikimini_arwiki2    |
| wikimini_enwiki     |
| wikimini_eswiki     |
| wikimini_eswiki2    |
| wikimini_frwiki     |
| wikimini_frwiki2    |
| wikimini_itwiki     |
| wikimini_itwiki2    |
| wikimini_labwiki    |
| wikimini_labwiki2   |
| wikimini_stockwiki  |
| wikimini_stockwiki2 |
| wikimini_svwiki     |
| wikimini_svwiki2    |
| wikimini_testwiki   |
| wikimini_testwiki2  |
+---------------------+
21 rows in set (0.16 sec)

To dump everything note you may need ~30 minutes:

$ ./dump.sh wikimini
Thu Nov 26 11:43:45 CET 2020 - dumping wikimini_arwiki...
Thu Nov 26 11:43:45 CET 2020 - dumping wikimini_arwiki2...
Thu Nov 26 11:43:46 CET 2020 - dumping wikimini_enwiki...
Thu Nov 26 11:43:47 CET 2020 - dumping wikimini_eswiki...
Thu Nov 26 11:48:02 CET 2020 - dumping wikimini_eswiki2...
Thu Nov 26 11:56:18 CET 2020 - dumping wikimini_frwiki...
Thu Nov 26 11:57:20 CET 2020 - dumping wikimini_frwiki2...
Thu Nov 26 11:58:08 CET 2020 - dumping wikimini_itwiki...
Thu Nov 26 12:05:41 CET 2020 - dumping wikimini_itwiki2...
Thu Nov 26 12:07:05 CET 2020 - dumping wikimini_labwiki...
Thu Nov 26 12:07:09 CET 2020 - dumping wikimini_labwiki2...
Thu Nov 26 12:07:13 CET 2020 - dumping wikimini_stockwiki...
Thu Nov 26 12:07:50 CET 2020 - dumping wikimini_stockwiki2...
Thu Nov 26 12:08:10 CET 2020 - dumping wikimini_svwiki...
Thu Nov 26 12:08:23 CET 2020 - dumping wikimini_svwiki2...
Thu Nov 26 12:08:27 CET 2020 - dumping wikimini_testwiki...
Thu Nov 26 12:08:28 CET 2020 - dumping wikimini_testwiki2...

This is their related size:

$ du -h *.sql
500K	wikimini_arwiki2.sql
380K	wikimini_arwiki.sql
1.4M	wikimini_enwiki.sql
16G	wikimini_eswiki2.sql
16G	wikimini_eswiki.sql
1.1G	wikimini_frwiki2.sql
1.1G	wikimini_frwiki.sql
2.8G	wikimini_itwiki2.sql
2.9G	wikimini_itwiki.sql
33M	wikimini_labwiki2.sql
33M	wikimini_labwiki.sql
480M	wikimini_stockwiki2.sql
491M	wikimini_stockwiki.sql
83M	wikimini_svwiki2.sql
82M	wikimini_svwiki.sql
1.1M	wikimini_testwiki2.sql
1.1M	wikimini_testwiki.sql
NOTE: Wikimini's eswiki itself is 16G.
$ du -sh /home/datas/www/Wikimini/wikimini.org
23G	/home/datas/www/Wikimini/wikimini.org

$ du -sh /home/datas/www/Wikimini/wikimini.org/*
12G	/home/datas/www/Wikimini/wikimini.org/httpdocs
9.6G	/home/datas/www/Wikimini/wikimini.org/httpdocs_1.25
1.3G	/home/datas/www/Wikimini/wikimini.org/logs

Transitional URLs

A list of URLs that now should redirect to the related https://wikimini.org sub-domain with a HTTP/1.1 301 Moved Permanently:

Event Timeline

valerio.bozzolan triaged this task as Medium priority.

Note that actually we cannot proceed because in WMCH members2 we have:

$ df -lh
Size  Used Avail Use% Mounted on
28G   8.4G   18G  33% /

And in WMCH demo:

$ df -lh
Size  Used Avail Use% Mounted on
28G   3.0G   23G  12% /

I've migrated everything with this dummy script:

1#!/bin/bash
2#
3# WIKIMINI MIGRATION SCRIPT
4#
5#########################################################
6#
7# Author
8# https://phabricator.wikimedia.org/p/valerio.bozzolan/
9# License
10# https://www.wikidata.org/wiki/Q152481
11# See
12# https://phabricator.wikimedia.org/T268292
13#
14#########################################################
15
16# die in case of error
17set -e
18
19#
20# Import an .sql file
21#
22# @param string name
23#
24import_wikimini() {
25
26 local db=wikimini_"$1"
27
28 # no database no party
29 if mysql "${db}" >/dev/null 2>&1 </dev/null; then
30 echo "skip existing database $db"
31 return
32 fi
33
34 # create missing database if not exists
35 #mysql <<< "CREATE DATABASE IF NOT EXISTS \`$db\`;"
36
37 # dump the database remotely, then stream everything locally and import just in place - WHOA!
38 message "transfering and importing database $db"
39 ssh -C wikimini.org mysqldump --force --insert-ignore --single-transaction --disable-keys -- "$db" | mysql "$db"
40}
41
42#
43# Print a stupid message to stdout with a date
44#
45# @param string msg..
46#
47message() {
48 echo "[$(date)] $@"
49}
50
51# transfer the remote wiki files and images in our local place
52message "fetching remote files and images"
53
54rsync --archive --progress --compress \
55 wikimini.org:/home/datas/www/Wikimini/wikimini.org/httpdocs/wiki/images/ \
56 /var/www/wikimini.org/www/wiki/images
57
58# import each single wiki
59import_wikimini arwiki
60import_wikimini enwiki
61import_wikimini svwiki
62import_wikimini frwiki
63import_wikimini itwiki
64import_wikimini labwiki
65import_wikimini stockwiki
66import_wikimini testwiki
67
68# biggest
69import_wikimini eswiki
70
71# in use? maybe not
72#import_wikimini arwiki2
73#import_wikimini eswiki2
74#import_wikimini frwiki2
75#import_wikimini itwiki2
76#import_wikimini labwiki2
77#import_wikimini stockwiki2
78#import_wikimini svwiki2
79#import_wikimini testwiki2

With this script I'm able to import everything on the fly.

Then I've deployed some versioned configurations:

cd /etc/apache2/sites-available
ln --symbolic /etc/wmch-infrastructure/servers/demo/projects/wikimini/apache2/sites/* .
a2ensite ch0*
apache2ctl graceful

Deploying few Let's Encrypt certificates:

$ letsencrypt certonly --webroot --webroot-path=/var/www/wikimini.org/startpage -d wikimini.wikimedia.ch
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for wikimini.wikimedia.ch
Using the webroot path /var/www/wikimini.org/startpage for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/wikimini.wikimedia.ch/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/wikimini.wikimedia.ch/privkey.pem
   Your cert will expire on 2021-03-17. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le
$ certbot certonly --webroot --webroot-path=/var/www/wikimini.org/www/w -d fr.wikimini.wikimedia.ch
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for fr.wikimini.wikimedia.ch
Using the webroot path /var/www/wikimini.org/www/w for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/fr.wikimini.wikimedia.ch/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/fr.wikimini.wikimedia.ch/privkey.pem
   Your cert will expire on 2021-03-17. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Now online at least the French version and their landing page:

https://wikimini.wikimedia.ch/

https://fr.wikimini.wikimedia.ch/

Soon this Task will be pinged from the related commit in this repo:

Continuing:

$ certbot certonly --webroot --webroot-path=/var/www/wikimini.org/www/w -d stock.wikimini.wikimedia.ch
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/ar.wikimini.org.conf with version 0.31.0 of Certbot. This might not work.
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/en.wikimini.org.conf with version 0.31.0 of Certbot. This might not work.
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/es.wikimini.org.conf with version 0.31.0 of Certbot. This might not work.
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/fr.wikimini.org.conf with version 0.31.0 of Certbot. This might not work.
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/it.wikimini.org-0001.conf with version 0.31.0 of Certbot. This might not work.
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/lab.wikimini.org.conf with version 0.31.0 of Certbot. This might not work.
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/stock.wikimini.org.conf with version 0.31.0 of Certbot. This might not work.
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/sv.wikimini.org.conf with version 0.31.0 of Certbot. This might not work.
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/wikimini.org.conf with version 0.31.0 of Certbot. This might not work.
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for stock.wikimini.wikimedia.ch
Using the webroot path /var/www/wikimini.org/www/w for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/stock.wikimini.wikimedia.ch/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/stock.wikimini.wikimedia.ch/privkey.pem
   Your cert will expire on 2021-04-21. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

$ a2ensite ch0-wikimedia-wikimini-stock-ssl.conf
$ apachectl graceful

Now online:

https://stock.wikimini.wikimedia.ch/wiki/Main_Page

I've disabled the Extension:WhosOnline that was causing unuseful overhead. Note that now we have Matomo (T268331).

That extension was also the cause of a couple of exceptions in every page when in read-only mode.

valerio.bozzolan moved this task from Working On to Recent Changes on the WMCH-Infrastructure board.
$ a2ensite ch0-wikimedia-wikimini-en000-txt
$ a2ensite ch0-wikimedia-wikimini-ar000-txt
$ a2ensite ch0-wikimedia-wikimini-lab00-txt
$ apachectl graceful

$ certbot certonly --webroot --webroot-path=/var/www/wikimini.org/www/w -d en.wikimini.wikimedia.ch
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for en.wikimini.wikimedia.ch
Using the webroot path /var/www/wikimini.org/www/w for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/en.wikimini.wikimedia.ch/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/en.wikimini.wikimedia.ch/privkey.pem
   Your cert will expire on 2021-04-22. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https:/

$ certbot certonly --webroot --webroot-path=/var/www/wikimini.org/www/w -d ar.wikimini.wikimedia.ch
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for ar.wikimini.wikimedia.ch
Using the webroot path /var/www/wikimini.org/www/w for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/ar.wikimini.wikimedia.ch/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/ar.wikimini.wikimedia.ch/privkey.pem
   Your cert will expire on 2021-04-22. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

$ # certbot certonly --webroot --webroot-path=/var/www/wikimini.org/www/w -d lab.wikimini.wikimedia.ch
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for lab.wikimini.wikimedia.ch
Using the webroot path /var/www/wikimini.org/www/w for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/lab.wikimini.wikimedia.ch/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/lab.wikimini.wikimedia.ch/privkey.pem
   Your cert will expire on 2021-04-22. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

$ a2ensite ch0-wikimedia-wikimini-en000-ssl
$ a2ensite ch0-wikimedia-wikimini-ar000-ssl
$ a2ensite ch0-wikimedia-wikimini-lab00-ssl

$ apachectl graceful

Tried an upload but:

Error creating thumbnail: /bin/bash: /usr/bin/convert: No such file or directory
Error code: 127

Fixed with:

apt install imagemagick

There were some issues updating references in this "swiffy" banner:

image.png (290×841 px, 76 KB)

I've manually updated this blob:

https://en.wikimini.wikimedia.ch//w/skins/Wikimini/resources/js/fr/swiffy-object.js

We are ready to internally redirect the whole traffic to this maintenance page during the migration:

https://wikimini.wikimedia.ch/maintenance/

For the TV series "don't go on vacation after a migration" here an interesting showcase of very interesting things occurred yesterday:

(To be wikificated)

AWC Forum broken

Issue:

Feb 19 15:47:54.841049 2021] [php7:warn] [pid 9540] [client 207.46.13.13:26854] PHP Warning:  preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /../extensions-external/awc-fr/forums/includes/gen_funk.php on line 70

Fix:

1diff --git a/gen_funk.php b/gen_funk.php
2index e71285d..1d5e769 100644
3--- a/gen_funk.php
4+++ b/gen_funk.php
5@@ -68,9 +68,16 @@ There are a few places in the forum where text being passed should not be phase
6 so they are base64_encode where needed and then this function will convert them all back in the end.
7 */
8
9-
10- $text = preg_replace('/'."@$id@".'([0-9a-zA-Z\\+\\/]+=*)'."@$id@".'/e', 'base64_decode("$1")', $text );
11-
12+ // hot patch
13+ $text = preg_replace_callback(
14+ '/'."@$id@".'([0-9a-zA-Z\\+\\/]+=*)'."@$id@".'/',
15+ function ($matches) {
16+ return base64_decode($matches[1]);
17+ },
18+ $text
19+ );
20+
21+ return $text;
22 }
23
24 function awcf_hide_code2($post, $todo = 'hide'){

ResourceLoaderULSJsonMessageModule broken

Issue:

[Fri Feb 19 15:52:34.498611 2021] [php7:warn] [pid 9732] [client 88.164.92.163:18707] PHP Warning:  Declaration of ::getDependencies() should be compatible with ResourceLoaderModule::getDependencies(?ResourceLoaderContext $context = NULL) in /var/www/wikimini.org/www/w/extensions-external/UniversalLanguageSelector/includes/ResourceLoaderULSJsonMessageModule.php on line 0, referer: https://fr.wikimini.org/wiki/Ara_Pacis

Fix:

extensions-external/UniversalLanguageSelector/includes/ResourceLoaderULSJsonMessageModule.php
diff --git a/ResourceLoaderULSJsonMessageModule.php b/ResourceLoaderULSJsonMessageModule.php
index 16e1c0c..537b574 100644
--- a/ResourceLoaderULSJsonMessageModule.php
+++ b/ResourceLoaderULSJsonMessageModule.php
@@ -18,7 +18,7 @@ class ResourceLoaderULSJsonMessageModule extends ResourceLoaderModule {
         * Declares the core ext.uls.i18n module as a dependency.
         * @return string[] Module names.
         */
-       function getDependencies() {
+       function getDependencies( ResourceLoaderContext $context = NULL ) {
                return array( 'ext.uls.i18n' );
        }

Related topic:

https://www.mediawiki.org/wiki/Topic:V8ex91ia0qo7c0y1


Error writing to CDB file

The previous issues were causing 30MB/s of I/O in the log file causing clogging of the disk and causing this unrelated exception:

Fatal error: Uncaught Cdb\Exception: Error writing to CDB file

Related discussion:
https://www.mediawiki.org/wiki/Topic:Tdz6lu5ihh9uakft

Cannot change session id when session is active

Issue:

[Sat Feb 20 09:02:59.001635 2021] [php7:warn] [pid 29269] [client 107.173.248.102:11522] PHP Warning:  session_id(): Cannot change session id when session is active in /var/www/wikimini.org/www/w/includes/Setup.php on line 787, referer: https://it.wikimini.org/w/

Fix:

T213489: 'Warning: session_id(): Cannot change session id when session is active' intermittently on php7.3

AWC's misc.php causing Illegal string offset

Issue:

[Sat Feb 20 09:04:53.229672 2021] [php7:warn] [pid 29536] [client 157.90.125.126:54234] PHP Warning:  Illegal string offset 'type' in /var/www/wikimini.org/www/w/extensions-external/awc-fr/forums/misc.php on line 179
[Sat Feb 20 09:04:53.229751 2021] [php7:warn] [pid 29536] [client 157.90.125.126:54234] PHP Warning:  Illegal string offset 'where' in /var/www/wikimini.org/www/w/extensions-external/awc-fr/forums/misc.php on line 18

https://fr.wikimini.org/w/api.php?action=visualeditor&format=json&paction=parse&page=Shakira&uselang=fr

{
  "error": {
    "code": "(curl error: 28) Timeout was reached",
    "info": "docserver-http-error: (curl error: 28) Timeout was reached",
    "*": "See https://fr.wikimini.org/w/api.php for API usage"
  }
}

Caused by port :8000 blocked by hardware firewall.

User sessions:

There is still a problem with the user sessions though. While editing an article, users randomly get disconnected and cannot submit its modification. This occurs after a few minutes. Users also get immediately disconnected while switching from a wiki to another (ex: from Wikimini FR to Wikimini Stock).

Less urgent issues:
The picture of the day doesn't work anymore. On both Wikimini FR and Wikimini SV, the pictures and their caption where being automatically generated by two custom coded scripts located in this directory: https://fr.wikimini.org/w/pictures-of-the-day/. As far as I remember, two cron jobs were running these scripts daily. Maybe it's necessary to recreate these cron jobs on the new server and run a CHMOD 777 on the directories where the thumbnails are being saved.

It looks like the WhosOnline Extension was removed. We were using this extension to show the online users on the main page (only to the logged in users) as seen in the attached picture.
I performed a speed test on both the Google Page speed tool and GT Metrics, and unfortunately the results are quite bad (20/100 on Google Page Speed and F Score on GT Metrics). Both tools offer a few suggestions, and within these solutions "Enable text compression" seems to be the one that would have the higher impact. I don´t know how easy that would. On the previous server, we were using xcache as a caching system. I don't know if it's still being used? Once in the past, I also tried to use the caching functionalities of Cloudlfare, but there were some problems on the wiki (recent changes not updating, error 500 on create account page...). This is the reason why we currently use Cloudlfare for DNS only.
Thank you again for your efforts and attention!

  • Laurent Jauquier
valerio.bozzolan updated the task description. (Show Details)
valerio.bozzolan moved this task from Working On to Recent Changes on the WMCH-Infrastructure board.

Also suppressed this:

[Mon Mar 08 14:30:57.361003 2021] [php7:warn] [pid 28124] [client 127.0.0.1:38218] PHP Warning:  count(): Parameter must be an array or an object that implements Countable in /var/www/wikimini.org/www/w/includes/api/ApiQueryBase.php on line 266