Page MenuHomePhabricator

DicoAdo.org: import legacy data from Google Analytics into WMCH Matomo to be able to drop Big-G
Closed, ResolvedPublic

Assigned To
Authored By
valerio.bozzolan
Apr 9 2022, 6:44 PM
Referenced Files
F35525224: Screenshot_2022_09_19_213913.png
Sep 19 2022, 7:44 PM
F35525229: Screenshot_2022_09_19_214151.png
Sep 19 2022, 7:44 PM
F35336380: Screenshot_2022_07_25_193022.png
Jul 25 2022, 5:31 PM
F35312054: Matomo Dashboard.png
Jul 8 2022, 1:52 PM
F35310527: Matomo Google Analytics Importer Plugin.png
Jul 7 2022, 9:21 AM
F35061321: Matomo Google Analytics Importer plugin.png
Apr 21 2022, 10:06 AM
Tokens
"Party Time" token, awarded by valerio.bozzolan."Like" token, awarded by DSwissK.

Description

For DicoAdo.org: check if we can import legacy data from Google Analytics into WMCH Matomo to be able to drop Big-G.

Interesting document:

  • install Google Analytics Importer plugin in WMCH Matomo
  • authorize at least one Google "test user" in Google Analytics
  • import Client Configuration
  • authorize the Importer (OAuth)
  • import data into Matomo

Details

Other Assignee
DSwissK

Event Timeline

This look like it can only be done using Matomo's side. We don't "export data from Google" and then give them to Matomo, Matomo fetches it itself. The first step is to install the relevant plugin.

valerio.bozzolan updated the task description. (Show Details)

No, we need to be superuser on Matomo apparently, we need to see system's parameters :/ . And @DSwissK will need to follow the tutorial in order to create an OAuth application in Google Analytics :-)

OK! Here a screenshot about what it's needed from our side (an "OAuth Client configuration file"):

Matomo Google Analytics Importer plugin.png (743×1 px, 76 KB)

Feel free to send it to me in private once generated.

Perfect. @DSwissK, you'll need to create an OAuth access on Google Analytics using the link mentionned above :-)
We can do this together if you want!

OK! Here a screenshot about what it's needed from our side (an "OAuth Client configuration file"):

Matomo Google Analytics Importer plugin.png (743×1 px, 76 KB)

Feel free to send it to me in private once generated.

@valerio.bozzolan you got a mail. :)

Thank you !

Amazing, thank you! I will try that ASAP (I think tomorrow). BTW happy liberation day!

https://en.wikipedia.org/wiki/Liberation_Day_(Italy)

Hey @valerio.bozzolan, any news about this ? :)
Kind regards.

I'm not sure I received a "OAuth access" via email. Can you please send that again?

(Maybe we do it faster on Telegram. I'm @bozzy)

Thank you so much

Yes, I did, but I think it's only valid 7 days.

I've done the procedure again and sent you an e-mail.

Thank you !

I've tried the "Authorize button" and it says this:

Error 403: access_denied
The developer hasn’t given you access to this app. It’s currently being tested and it hasn’t been verified by Google. If you think you should have access, contact the developer (epiney@<OMISSIS>).
More information
https://support.google.com/accounts/answer/3466521?p=app_notverified

I'm reading some documentation about this error.

Yeah! Now they say:

Make sure dicoado.org is trusted
Yes / No

I suggest to BigG to add, after Yes/no, also a button called "Absolutely yes, I trust DicoAdo.org and I would give them my house keys; unlike you, BigG in which I wouldn't even give you access to my fridge - now let me import everything and delete Google Analytics forever"

OK awesome now we just need to understand what to put there:

Matomo Google Analytics Importer Plugin.png (2×1 px, 363 KB)

Here the official documentation:

https://matomo.org/docs/google-analytics-importer/

I am reading this now.

I'd go with these values :

Start date : 17 Septembre 2016
Tracking ID : UA-49598591-1
Account ID : 49598591 (or maybe yours is different, with your gmail account ?)
Analytic View ID : 84248374

Yep and as end date we could try 2020-11-26 since our Matomo data starts at 2020-11-27 if I understand correctly

I mean 2022-04-04 since our data starts at 2022-04-05

I'm inclined to leave the start date empty since it says "The date to start importing from. By default, this is the day you created your GA web property." so the default could be nice enough.

I'm doing a pre-import backup and then start in 10 minutes.

valerio.bozzolan renamed this task from DicoAdo.org: check if we can import legacy data from Google Analytics into WMCH Matomo to be able to drop Big-G to DicoAdo.org: import legacy data from Google Analytics into WMCH Matomo to be able to drop Big-G.Jul 7 2022, 9:46 AM
valerio.bozzolan claimed this task.
valerio.bozzolan updated Other Assignee, added: DSwissK.
valerio.bozzolan updated the task description. (Show Details)

One thing that was non-obvious: the import creates another website.

https://github.com/matomo-org/plugin-GoogleAnalyticsImporter/issues/146#issuecomment-720164190

Having said that, the web interface is nice but I tried the command line way that is more verbose:

$ ./console googleanalyticsimporter:import-reports --property=UA-49598591-1 --view=84248374 --dates=2016-09-17,2022-04-04"
No account ID specified, assuming it is '49598591'.
Created new site with ID = 4.
WARNING [2022-07-07 10:16:48] 7755  Unable to import the 'More than 2 pages / visit' goal: unsupported goal type.
WARNING [2022-07-07 10:16:48] 7755  Importing this goal as a manually triggered goal. Metrics for this goal will be available, but tracking will not work for this goal in Matomo.
WARNING [2022-07-07 10:16:48] 7755  Unable to import the 'Check "about" page' goal: unknown goal match type, 'REGEX'.
WARNING [2022-07-07 10:16:48] 7755  Importing this goal as a manually triggered goal. Metrics for this goal will be available, but tracking will not work for this goal in Matomo.
WARNING [2022-07-07 10:16:48] 7755  Unable to import the 'Check Help pages' goal: unknown goal match type, 'REGEX'.
WARNING [2022-07-07 10:16:48] 7755  Importing this goal as a manually triggered goal. Metrics for this goal will be available, but tracking will not work for this goal in Matomo.
WARNING [2022-07-07 10:16:48] 7755  Unable to import the 'Visit "Create an account"' goal: unknown goal match type, 'REGEX'.
WARNING [2022-07-07 10:16:48] 7755  Importing this goal as a manually triggered goal. Metrics for this goal will be available, but tracking will not work for this goal in Matomo.
WARNING [2022-07-07 10:16:48] 7755  Unable to import the 'Check "Pour les profs" page' goal: unknown goal match type, 'REGEX'.
WARNING [2022-07-07 10:16:48] 7755  Importing this goal as a manually triggered goal. Metrics for this goal will be available, but tracking will not work for this goal in Matomo.
WARNING [2022-07-07 10:16:48] 7755  Unable to import the 'Ajouter un mot' goal: unknown goal match type, 'REGEX'.
WARNING [2022-07-07 10:16:48] 7755  Importing this goal as a manually triggered goal. Metrics for this goal will be available, but tracking will not work for this goal in Matomo.
WARNING [2022-07-07 10:16:48] 7755  Unable to import the 'Edit with pageform' goal: unknown goal match type, 'REGEX'.
WARNING [2022-07-07 10:16:48] 7755  Importing this goal as a manually triggered goal. Metrics for this goal will be available, but tracking will not work for this goal in Matomo.
WARNING [2022-07-07 10:16:48] 7755  Unable to import the 'Edit wikicode' goal: unknown goal match type, 'REGEX'.
WARNING [2022-07-07 10:16:48] 7755  Importing this goal as a manually triggered goal. Metrics for this goal will be available, but tracking will not work for this goal in Matomo.
INFO [2022-07-07 10:16:48] 7755  Using existing custom variable slots.
Importing the following date ranges in order: 2016-09-17,2022-04-04
Importing reports for date range 2016-09-17 - 2022-04-04 from GA view 84248374.
INFO [2022-07-07 10:16:48] 7755  Importing data for GA View 84248374 for date 2016-09-17...

Apart from the warnings, then after some minutes it exits with:

Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

Maybe as you said it's just something related to my account ID soon. I will retry after 14:30.

INFO [2022-07-07 15:21:15] 29821  Daily rate limit reached, try again tomorrow. (Note: GA by default sets a daily limit on the number of API requests made each day to 50000. It looks like you've reached this limit. Continue the import tomorrow.)

Shit. BigG: you may have won this battle but you will not win the war.

RIP! No problem we'll continue tomorrow. Thanks for your time!

Le jeu. 7 juil. 2022 à 17:25, valerio.bozzolan <
no-reply@phabricator.wikimedia.org> a écrit :

valerio.bozzolan added a comment. View Task
https://phabricator.wikimedia.org/T305789

INFO [2022-07-07 15:21:15] 29821 Daily rate limit reached, try again tomorrow. (Note: GA by default sets a daily limit on the number of API requests made each day to 50000. It looks like you
've reached this limit. Continue the import tomorrow.)

Shit. BigG: you may have won this battle but you will not win the war.

*TASK DETAIL*
https://phabricator.wikimedia.org/T305789

*EMAIL PREFERENCES*
https://phabricator.wikimedia.org/settings/panel/emailpreferences/

*To: *valerio.bozzolan
*Cc: *DSwissK, Raphoraph, valerio.bozzolan

It seems the Google Analytics "data sucker" is working perfectly but it's heavily API rate-limited. It seems its extimated end date is in... 22 days since today.

Matomo Google Analytics Importer Plugin Import Jobs rate limited.png (374×1 px, 54 KB)

So I'd say: let's wait! Yeeh!

You should have enough privileges to see the import progress, playing with data ranges. It's not clear to me why it starts from 17 September 2016 but... let's wait.

Matomo Dashboard.png (554×1 px, 138 KB)

Wow,I hope the credential don't deprecate during that time...

And yes, you gave me enough rights to see the data, thank you !

The website was created on 17 September 2016 so that's all good, I guess.

OK folks it seems Matomo's import arrived at November 2018. Now the credentials are expired but I renewed them successfully. Still importing!

Screenshot_2022_07_25_193022.png (622×1 px, 191 KB)

Nice, you're a wizard! 😉

Thank you.

Thank you for the follow-up!

@DSwissK, no, he's a sysadmin :D

By the way, I did not take note of the password to access Dicoado's Matomo… Could you reset it for me please? :3 I have 2FA, I can give you an access code if needed.

Whoa, it seems Matomo has concluded sucking all data from Google Analytics!

Screenshot_2022_09_19_214151.png (862×1 px, 247 KB)

Screenshot_2022_09_19_213913.png (226×1 px, 45 KB)

Note the "Finished" flag from the above report.

sgtm

Let's mark as resolved. Feel free to discuss!

Awesome! Now needs to remove GA completely.
DSwissK perhaps has some question about some difference between stats of Matomo and stats of GA, feel free to ask @DSwissK.

Finally could you reset my password for that matomo instance please? I've lost it…

For the password reset let's discuss here: https://meta.wikimedia.org/wiki/User_talk:Raphoraph I need to be 1000% secure about a small thing. Thank and sorry for this.