Page MenuHomePhabricator

iOS traffic data is not available on Piwik since Feb 20, 2018
Closed, ResolvedPublic5 Estimate Story Points

Description

We don't see any iOS traffic data on Piwik since Feb 20, 2018, although real-time visitor data is still available.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 28 2018, 10:46 PM

Also, some other sites have suspiciously low counts as of the last couple days

There are plenty hits:

root@bohrium:/var/log/apache2# zcat other_vhosts_access.log.2.gz | grep -i iOs | wc -l
1104068

Nuria added a comment.EditedMar 1 2018, 4:46 AM

I wonder if the archive cron (which i think is running once a day) has not run due to machine failures for couple times and now it cannot run due to backlog of data. Maybe we need to setup crons per site once a day rather than having 1 cron processing all sites

Nuria added a comment.Mar 1 2018, 4:50 AM

Executed archiving by hand and got the following for website "3" which is iOS
INFO [2018-03-01 04:37:22] - Will invalidate archived reports for 2018-02-28 for following websites ids: 3 on

elukey added a comment.Mar 1 2018, 7:18 AM

Executed the following:

elukey@bohrium:/var/log/piwik$
elukey@bohrium:/var/log/piwik$ for el in {20..28}; do sudo -u www-data /usr/share/piwik/console core:invalidate-report-data --dates=2018-02-$el --sites=3; done
Invalidating day periods in 2018-02-20 [segment = ]...
Invalidating week periods in 2018-02-20 [segment = ]...
Invalidating month periods in 2018-02-20 [segment = ]...
Invalidating year periods in 2018-02-20 [segment = ]...
Invalidating day periods in 2018-02-21 [segment = ]...
Invalidating week periods in 2018-02-21 [segment = ]...
Invalidating month periods in 2018-02-21 [segment = ]...
Invalidating year periods in 2018-02-21 [segment = ]...
Invalidating day periods in 2018-02-22 [segment = ]...
Invalidating week periods in 2018-02-22 [segment = ]...
Invalidating month periods in 2018-02-22 [segment = ]...
Invalidating year periods in 2018-02-22 [segment = ]...
Invalidating day periods in 2018-02-23 [segment = ]...
Invalidating week periods in 2018-02-23 [segment = ]...
Invalidating month periods in 2018-02-23 [segment = ]...
Invalidating year periods in 2018-02-23 [segment = ]...
Invalidating day periods in 2018-02-24 [segment = ]...
Invalidating week periods in 2018-02-24 [segment = ]...
Invalidating month periods in 2018-02-24 [segment = ]...
Invalidating year periods in 2018-02-24 [segment = ]...
Invalidating day periods in 2018-02-25 [segment = ]...
Invalidating week periods in 2018-02-25 [segment = ]...
Invalidating month periods in 2018-02-25 [segment = ]...
Invalidating year periods in 2018-02-25 [segment = ]...
Invalidating day periods in 2018-02-26 [segment = ]...
Invalidating week periods in 2018-02-26 [segment = ]...
Invalidating month periods in 2018-02-26 [segment = ]...
Invalidating year periods in 2018-02-26 [segment = ]...
Invalidating day periods in 2018-02-27 [segment = ]...
Invalidating week periods in 2018-02-27 [segment = ]...
Invalidating month periods in 2018-02-27 [segment = ]...
Invalidating year periods in 2018-02-27 [segment = ]...
Invalidating day periods in 2018-02-28 [segment = ]...
Invalidating week periods in 2018-02-28 [segment = ]...
Invalidating month periods in 2018-02-28 [segment = ]...
Invalidating year periods in 2018-02-28 [segment = ]...

https://matomo.org/faq/how-to/faq_59/

Side note: now Piwik is called Matomo :)

elukey added a comment.Mar 1 2018, 8:32 AM

Good news: Nuria was right, the archiver was adding missing data after invalidation.

Bad news: the extra IO (probably) triggered the Ganeti bug that causes the underlying host to freeze (bohrium, on which Piwik runs, is a Ganeti Virtual Machine) so I got only data up to the 22rd:

Added some documentation: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Piwik#Invalidate_old_reports

Seems better now (archiver still running):

Nuria added a comment.Mar 1 2018, 4:01 PM

Super thanks @elikey. @chelsyx data shoudl be back but please have in mind that piwik is not a very reliable data store, nor does it have teh availability gurantees of eventlogging. It works well for small sites but IOs data is already pushing the limits.

elukey added a comment.Mar 1 2018, 4:14 PM

Interesting stat: it took ~5h to archive the iOS data for the past week :D

Archived website id = 3, 4 API requests, Time elapsed: 19351.548s [2/12 done]
Nuria edited projects, added Analytics-Kanban; removed Analytics.Mar 1 2018, 5:07 PM
Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.

Thanks @Nuria and @elukey !

@JMinor, considering the unreliability of piwik, we should figure out a way with the Analytics team to back up the data, in case outage like this happen again.

Nuria added a comment.Mar 1 2018, 6:16 PM

Data is backed up, now, piwik is a tool mean to be used for low traffiqued sites, otherwise it just cannot handle it. In this case IOS data is getting too large. I think @JMinor is aware of this fact.

@Nuria I was talking about the outage on Nov 23 2017, which the data cannot be recovered. Is there any risk that this kind of outage happen again?

JMinor added a comment.Mar 1 2018, 6:52 PM

Yes, per @Nuria this is a low priority system. I appreciate that gaps in data are problematic for analysis, but given that we plan to move away from it over this year, I think we're sufficiently backed up for now.

Thanks @elukey and @Nuria for the reboot/resurrection.

Nuria added a comment.Mar 2 2018, 5:50 AM

I invalidated the rest of the sites and run the archiver so all sites should be good now.

Nuria set the point value for this task to 5.Mar 2 2018, 5:50 AM
Nuria moved this task from In Progress to Done on the Analytics-Kanban board.
Nuria closed this task as Resolved.Mar 2 2018, 9:36 PM
Nuria reopened this task as Open.Mar 5 2018, 6:27 PM
Nuria closed this task as Resolved.