Description
Details
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
Upgrade the piwik module to matomo | operations/puppet | production | +30 -25 |
Event Timeline
I tested today in labs an upgrade from the current version 2.17 to 3.2.1, the latest stable before the rebrand to matomo. Everything worked fine, but the update function will probably take 1/2 hours to make all the invasive database changes. It should be doable very soon, so we'll get a version that was released last December. The next step will be to upgrade to matomo as well.
Piwik maintenance just completed, the website is back in service with the new 3.2.1 version.
The last version of Matomo (the new Piwik name) contains a super interesting feature, namely use TLS for connections to the database. This is not important now that we co-locate Piwik with its db on bohrium, but for future usage (for example, if we'll want to move the db out of bohrium).
I tried to install it in labs, this is what I get:
elukey@piwik:~$ sudo dpkg -i matomo_3.5.1-1_all.deb Selecting previously unselected package matomo. (Reading database ... 60845 files and directories currently installed.) Preparing to unpack matomo_3.5.1-1_all.deb ... * Piwik is now Matomo. Matomo configuration files are stored in /etc/matomo/ * Piwik has been previously installed on this system and its configuration folder * remains. We're moving Piwik configuration folder from '/etc/piwik' to '/etc/piwik.orig' * so this package can install correctly. * Review the configuration changes accordingly. Unpacking matomo (3.5.1-1) ... Setting up matomo (3.5.1-1) ... Enabling conf matomo. To activate the new configuration, you need to run: service apache2 reload * Check Matomo web configuration in /etc/apache2/conf-available/matomo.conf * Copying Piwik configuration (/etc/piwik.orig/config.ini.php) to Matomo configuration folder (/etc/matomo/) * Matomo main configuration files are stored in /etc/matomo/ * Matomo crontab file is stored in /etc/cron.d/matomo-archive * When installing/using matomo, make sure you disable any ads blocker * To finish Matomo's upgrade, please run: /usr/bin/php /usr/share/matomo/console core:update
And then, there are db upgrades to do:
Executing CREATE TABLE `piwik_privacy_logdata_anonymizations` (`idlogdata_anonymization` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, `idsites` TEXT NULL DEFAULT NULL, `date_start` DATETIME NOT NULL, `date_end` DATETIME NOT NULL, `anonymize_ip` TINYINT(1) UNSIGNED NOT NULL DEFAULT 0, `anonymize_location` TINYINT(1) UNSIGNED NOT NULL DEFAULT 0, `anonymize_userid` TINYINT(1) UNSIGNED NOT NULL DEFAULT 0, `unset_visit_columns` TEXT NOT NULL DEFAULT '', `unset_link_visit_action_columns` TEXT NOT NULL DEFAULT '', `output` MEDIUMTEXT NULL DEFAULT NULL, `scheduled_date` DATETIME NULL, `job_start_date` DATETIME NULL, `job_finish_date` DATETIME NULL, `requester` VARCHAR(100) NOT NULL DEFAULT '', PRIMARY KEY ( `idlogdata_anonymization` )) ENGINE=InnoDB DEFAULT CHARSET=utf8;... Done. [1 / 6] Executing ALTER TABLE `piwik_privacy_logdata_anonymizations` ADD INDEX index_job_start_date (`job_start_date`);... Done. [2 / 6] Executing ALTER TABLE `piwik_plugin_setting` ADD COLUMN `json_encoded` TINYINT UNSIGNED NOT NULL DEFAULT 0;... Done. [3 / 6] Executing ALTER TABLE `piwik_site_setting` ADD COLUMN `json_encoded` TINYINT UNSIGNED NOT NULL DEFAULT 0;... Done. [4 / 6] Executing ALTER TABLE `piwik_log_visit` MODIFY COLUMN `location_region` char(3) DEFAULT NULL;... Done. [5 / 6] Executing ALTER TABLE `piwik_log_conversion` MODIFY COLUMN `location_region` char(3) DEFAULT NULL;... Done. [6 / 6]
They might take a while since piwik_log_visit is involved, but surely less than the last time (~8h).
Just to be super sure, triple checked that the matomo's tracking JS code didn't change:
<!-- Matomo --> <script type="text/javascript"> var _paq = _paq || []; /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { var u="//localhost:1080/"; _paq.push(['setTrackerUrl', u+'piwik.php']); _paq.push(['setSiteId', '1']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s); })(); </script> <!-- End Matomo Code -->
Change 442806 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Upgrade the piwik module to matomo
Piwik is currently archiving tables, so I am going to postpone the maintenance until it finishes.
Change 442806 merged by Elukey:
[operations/puppet@production] Upgrade the piwik module to matomo
Mentioned in SAL (#wikimedia-operations) [2018-06-28T14:46:40Z] <elukey> upgrade piwik 3.2.1 to matomo (new name/package) 3.5.1 - T192298
Also updated https://wikitech.wikimedia.org/wiki/Analytics/Systems/Piwik with the downtimes
Going back to in progress, some issues are coming up after the upgrade:
- Nuria had to restart Matomo (apachectl restart) due to slowness and timeouts. I can see the action in auth.log but not specific errors in the logs, will try to check latencies in apache logs as well:
Jun 28 18:03:21 bohrium sudo: nuria : TTY=pts/0 ; PWD=/usr/share/matomo ; USER=root ; COMMAND=/etc/init.d/apache2 stop Jun 28 18:03:26 bohrium sudo: nuria : TTY=pts/0 ; PWD=/usr/share/matomo ; USER=root ; COMMAND=/etc/init.d/apache2 start
- There are periodical/recurrent errors in the apache error log:
[Fri Jun 29 16:43:45.560500 2018] [:error] [pid 30120] [client 10.64.32.103:13001] Error in Matomo (tracker): Custom timestamp is 962307 seconds old, requires &token_auth...
If I try to correlate it with an access log request, I can see a big POST that looks from iOS with a weird timestamp:
url?QUERY_STRING_OMITTED&cdt=2018-06-28%2014:49:57
UI: WikipediaApp/5.7.2.1267 (iPhone OS 9.3.5; Tablet)
A bit weird..
It happens even now, and httpd returns application/x-httpd-php/400. The requests are look the same, GET with long query strings.. (not all the same client IPs).
Let's keep an eye on this, not sure if it is a new bug or feature :D
The only error i see on ui is: "curl_exec: Failed to connect to plugins.matomo.org port 443: Connection timed out. Hostname requested was: plugins.matomo.org"
In /var/log/apache/error.log there are still errors like:
[Tue Jul 03 05:52:59.776050 2018] [:error] [pid 5552] [client 10.64.32.103:7423] Error in Matomo (tracker): Custom timestamp is 283052 seconds old, requires &token_auth...
That corresponds to application/x-httpd-php/400 errors for UA WikipediaApp* (various versions). These agents are doing simple GET requests, so not POSTing events...