Page MenuHomePhabricator

Move usage logs from Sqlite to MariaDB
Open, Needs TriagePublic

Description

This ticket is a Phabricator placeholder for https://github.com/wsexport/tool/issues/173

Event Timeline

Restricted Application added a project: Community-Tech. · View Herald TranscriptJun 19 2019, 12:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

A preliminary patch, to make it possible to have custom config for tests, is https://github.com/wsexport/tool/pull/185 — it's ready for review.

ifried closed this task as Resolved.Jul 15 2019, 10:24 PM
Samwilson reopened this task as Open.Jul 15 2019, 10:27 PM

This issue is not yet resolved. https://github.com/wsexport/tool/pull/184 still needs to be reviewed and merged.

aezell added a subscriber: aezell.Jul 15 2019, 10:31 PM

This was my bad. I saw that 185 was merged. Thanks for clearing it up.

ifried closed this task as Resolved.Wed, Aug 14, 7:19 PM
Samwilson reopened this task as Open.Wed, Aug 14, 11:48 PM

Sorry, this is still waiting for review (PR184). I should have made sure it was in the right column.

This has been around for quite a while. What's the barrier to review? What's the risk in just moving forward with it as is?

Should I just go ahead with it? Am happy to.

The migration isn't that fraught, because if it goes wrong we'll still have the original data files and can revert to the Sqlite system. The main risks are a) not capturing stats correctly after the migration, and b) not importing the old stats correctly. I've tested it and can't find any errors.

Migration complete. There are a few things I still want to clean up in the prod tool (old symlinks etc.) but everything about the switch to mariadb is done (both for prod and staging).

dom_walden added subscribers: ifried, dom_walden.EditedTue, Aug 20, 10:36 AM

On wsexport-test, I created a number of pdf/epubs/etc. and saw the statistics rise as appropriate.

I saw no suspect messages in the error logs.

Installing on my local VM following the new install instructions went smoothly.

There was a bug where the statistics for the final day of a month were not being displayed (https://github.com/wsexport/tool/pull/196). This is now fixed on wsexport-test (but not production yet).

I believe when importing the old statistics from sqlite we ignore any duplicates of rows which have the same language, ebook title and timestamp.

There are a small number of these duplicates in the old sqlite database, presumably where the same ebook was generated twice within a second. Or, perhaps the duplicates are a bug in our old statistics collection code (I have not investigated this).

On production, in May, June and July we are missing about 800-1000 per month from a total of 75000-90000 per month.

@ifried @Samwilson I am not sure if we feel it worth fixing. It will only affect historic data, not data we collect from now on.