@Ragesoss Hello hello, this is just a reminder for the campaign team, I will ping you once we need the data on the following training modules for this WMDE campaign:

• Editieren Basiswissen: https://outreachdashboard.wmflabs.org/training/wikipedia-editieren/editieren-basiswissen

• Diskutieren mit Quellcode: https://outreachdashboard.wmflabs.org/training/wikipedia-editieren/diskutieren-basiswissen

Thanks! The campaign has just started and it will run for one month. We will not need the data before then.

@Christine_Domgoergen_WMDE Please check the tracking setup for this campaign, because

• until 23:00 CET, 2020/05/11 (today), there were none (zero) pageviews of the selected pages (see the 30 Days Wikipedia Challenge campaign doc) with the campaign tag attached (anything like WMDE_2020_challenge_ present in the uri_query filed of the wmf.webrequest table)

@Christine_Domgoergen_WMDE Confirmed:

• no pageviews received from the respective campaign tags on 2020/05/11 (first day of campaign), so
• no user registrations.

The data acquisition procedure now runs on a daily schedule (10:00 UTC) from stat1004.

@GoranSMilovanovic
Hi Goran, sorry I forgot to update you: we changed the timeline, the campaign will start this Thursday, May 14th. I will do a tracking test now and get back to you.

@GoranSMilovanovic
So I did a user registration through the following campaign tag and landingpage: https://de.wikipedia.org/wiki/Spezial:Benutzerkonto_anlegen?campaign=WMDE_2020_challenge_5
User: Test E-Mail Campaign 2020 1

Can you check if you have the page view and the user in the database? Thank you!

@Christine_Domgoergen_WMDE To be tested; it will take some time before the wmf.webrequest and event.serversideaccountcreation tables catch-up with the changes.

@GoranSMilovanovic Great, thank you. The link above was a e-mail link, so you should also have the page view right? But to make sure I produced another page view in Wikidata: https://www.wikidata.org/wiki/Wikidata:Introduction/de?campaign=WMDE_2020_challenge_10

Edit: and created a user account in Wikidata: Test E-Mail Campaign 2020 2

@Christine_Domgoergen_WMDE Yes yes right I had a moment of confusion - testing and reviewing two campaigns concurrently now... Anyways, we will wait for an hour or two to make sure the logging systems picked up the changes and test then. I will be reporting back here as soon as the tests are complete. Thank you!

@Christine_Domgoergen_WMDE Testing now.

@Christine_Domgoergen_WMDE Test results:

Pageviews

"","Tag","Page","Pageviews","date","campaign"
"1","?campaign=WMDE_2020_challenge_14","de.wikipedia.org/wiki/Special:MyPage/Artikelwerkstatt",1,2020-05-12,"Email_Campaign_Wikipedia_Challenge2020"
"2","?campaign=WMDE_2020_challenge_5","de.wikipedia.org/wiki/Spezial:Benutzerkonto_anlegen",1,2020-05-12,"Email_Campaign_Wikipedia_Challenge2020

So, we find your pageview described in T251524#6128649, but not the one described in T251524#6128691. Also, we find the pageview in the first row of the test result: de.wikipedia.org/wiki/Special:MyPage/Artikelwerkstatt with the WMDE_2020_challenge_14 tag attached but this pageview was not mentioned in this ticket before.

User registrations

Nothing yet as of the test user Test E-Mail Campaign 2020 2 on Wikidata. I have tested if the event.serversideaccountcreation schema catches up registrations from Wikidata, and it does with wiki="wikidatawiki" (14 user registered on Wikidata today, but none of them is you test user).

@GoranSMilovanovic
Great, thank you.

So, we find your pageview described in T251524#6128649, but not the one described in T251524#6128691.

So you couldn't find the one in Wikidata, right?

Also, we find the pageview in the first row of the test result: de.wikipedia.org/wiki/Special:MyPage/Artikelwerkstatt with the WMDE_2020_challenge_14 tag attached but this pageview was not mentioned in this ticket before.

yes I guess this was mine as well

Nothing yet as of the test user Test E-Mail Campaign 2020 2 on Wikidata. I have tested if the event.serversideaccountcreation schema catches up registrations from Wikidata, and it does with wiki="wikidatawiki" (14 user registered on Wikidata today, but none of them is you test user).

This is weird because the Test User is there: https://www.wikidata.org/wiki/User:Test_E-Mail_Campaign_2020_2

So it seems we have everything in Wikipedia but not the page views and user registrations in Wikidata, correct? Why could that be? Probably it would be good to test in Commons, Wikivoyage and Wiktionary as well, then. I will do some user registrations and let you know.

@Christine_Domgoergen_WMDE

So you couldn't find the one in Wikidata, right?

This one: https://www.wikidata.org/wiki/Wikidata:Introduction/de?campaign=WMDE_2020_challenge_10

yes I guess this was mine as well

Ok.

This is weird because the Test User is there: https://www.wikidata.org/wiki/User:Test_E-Mail_Campaign_2020_2

It could still be the case that the event.serversideaccountcreation table was not yet updated while I was testing.

Probably it would be good to test in Commons, Wikivoyage and Wiktionary as well, then. I will do some user registrations and let you know.

Please do.

@Christine_Domgoergen_WMDE Found it:

year    month   day     hour    campaign                userid  username                        webhost                 wiki
2020    5       12      9       WMDE_2020_challenge_5   3482942 Test E-Mail Campaign 2020 1     de.wikipedia.org        dewiki

I would say we are ready to go.

@GoranSMilovanovic

I produced page views and accounts in all projects:

Page View Wikivoyage: https://de.wikivoyage.org/w/index.php?title=Hauptseite&campaign=WMDE_2020_challenge_26
User Account Wikivoyage: Test E-Mail Campaign 2020 3

Page View Wiktionary: https://de.wiktionary.org/wiki/Wiktionary:Hauptseite?campaign=WMDE_2020_challenge_28
User Account Wikitonary: Test E-Mail Campaign 2020 4

Page View Commons: https://commons.wikimedia.org/wiki/Commons:Willkommen?campaign=WMDE_2020_challenge_7
User Account Commons: Test E-Mail Campaign 2020 5

@GoranSMilovanovic

I would say we are ready to go.

Great! Just to be sure can you check tomorrow if you can find the three new ones as well?

@Christine_Domgoergen_WMDE

Great! Just to be sure can you check tomorrow if you can find the three new ones as well?

Of course. I will take a look at the tables tonight.

@Christine_Domgoergen_WMDE Test results:

Pageviews I've found only the following until now for 2020/05/12:

de.wikivoyage.org/wiki/Hauptseite?campaign=WMDE_2020_challenge_26
de.wikipedia.org/wiki/Special:MyPage/Artikelwerkstatt?campaign=WMDE_2020_challenge_14
de.wiktionary.org/wiki/Wiktionary:Hauptseite?campaign=WMDE_2020_challenge_28
de.wikipedia.org/wiki/Spezial:Benutzerkonto_anlegen?campaign=WMDE_2020_challenge_5

User registrations

Nothing yet, but let's give it some time it seems that the event.serversideaccountcreation table takes some time to update.

@Christine_Domgoergen_WMDE

Confirming 2020/05/13 00:16 CET

• No pageviews for https://commons.wikimedia.org/wiki/Commons:Willkommen? registered.

@GoranSMilovanovic Thank you for checking! This is weird though... the URL is https://commons.wikimedia.org/wiki/Commons:Willkommen (without "?"), is this maybe the reason? If not, what other reasons could explain the problem here and can we do something about it?

Did you find the user registrations?

@Christine_Domgoergen_WMDE No no... the ? character is present only here, not in the code that is looking for pageviews.

Could you please check do the campaigns tags work appropriately with Commons pages? Because the code also filters out all pageviews without a campaign tag attached...

In the meanwhile, I will take a look at the pageviews code to see if there is something wrong there, because I do not seem to be able to grasp any campaign related Commons pageviews at all.

User registrations

"SELECT year, month, day, hour,  event.campaign, event.userId, event.userName, webhost, wiki 
      FROM event.serversideaccountcreation 
      WHERE 
        year = 2020 
        AND (month = 5 OR month = 6) 
        AND (event.campaign LIKE '%WMDE_2020_challenge_%');"

still returns only one user: Test E-Mail Campaign 2020 (already found in T251524#6129702).

@GoranSMilovanovic

Yes, when I click on a link with campaign tag leading to Commons the page is displayed normally: https://commons.wikimedia.org/?campaign=WMDE_2020_challenge_7 Is that what you mean by checking if the links work?

@Christine_Domgoergen_WMDE Yes, that means that the page is definitely tagged. So let me see now why can't I see it from the wmf.webrequest table.

Re-testing for user registrations now.

@Christine_Domgoergen_WMDE Here's what I have on the test users in T251524#6129729:

• found Test E-Mail Campaign 2020 3 on dewikivoyage;
• found Test E-Mail Campaign 2020 4 on dewiktionary;
• found Test E-Mail Campaign 2020 5 on commonswiki;

all in the event.serversideaccountcreation Hive table (where they were supposed to be),

however none of the records had any value in the event.campaign field - and that is the reason why the code failed to find these user registrations in my previous attempts (i.e. the code assumes that the user registration is accompanied by a relevant value in the event.campaign field; see T251524#6132612: event.campaign LIKE '%WMDE_2020_challenge_%).

In other words: the user registration logging somewhere fails to send the campaign tag to the event.serversideaccountcreation table.

@GoranSMilovanovic
Oh okay, what does this mean? Is there another way to get the data on user registrations? Or do we have to change something else in order to get the tag into the table? Is it the same problem we had in the last autumn banner campaign or is maybe the patch from Adam not in place anymore?

With the page views the tag is are sent to the table, right?

@Christine_Domgoergen_WMDE

Also, as of the following in T251524#6129203:

So it seems we have everything in Wikipedia but not the page views and user registrations in Wikidata, correct?

There is a user Test E-Mail Campaign 2020 2 found on wikidatawiki - and again, no value in the event.campaign field.
So my conclusions from T251524#6133796 hold for Wikidata registrations too.

Is there another way to get the data on user registrations?

Not that I know. As far as I understand all user registrations are now found in our Big Data storage, and that would be the event.serversideaccountcreation table used here.

Or do we have to change something else in order to get the tag into the table? Is it the same problem we had in the last autumn banner campaign or is maybe the patch from Adam not in place anymore?

That is a questions for people who have maintain the logging procedures code, not me. I only know that there is something called "the patch from Adam" @Addshore - but I have no idea what does the patch do.

With the page views the tag is are sent to the table, right?

I am now working on the pageviews, for this campaign and T251535#6133655 as well; as soon as I have something I will let you know. That shouldn't take long.

Unless someone have changed the code (the glue bit), my last memory was that it only runs on dewiki.

@Christine_Domgoergen_WMDE ^^ + I think I've found the bug that was preventing us from observing the Commons pageviews, testing now.

@GoranSMilovanovic @Addshore So if we want to track user registrations and editing behavior in wikidata, commons, wiktionary and wikivoyage the patch would need to be activated for those as well?

Yes
https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/master/includes/WikimediaEventsHooks.php#L601-L604

@Christine_Domgoergen_WMDE Found the Commons pageviews.

Now here is everything observed on 2020/05/12:

"","Tag","Page","Pageviews","date","campaign"
"1","?campaign=WMDE_2020_challenge_14","de.wikipedia.org/wiki/Special:MyPage/Artikelwerkstatt",1,2020-05-12,"Email_Campaign_Wikipedia_Challenge2020"
"2","?campaign=WMDE_2020_challenge_26","de.wikivoyage.org/wiki/Hauptseite",1,2020-05-12,"Email_Campaign_Wikipedia_Challenge2020"
"3","?campaign=WMDE_2020_challenge_28","de.wiktionary.org/wiki/Wiktionary:Hauptseite",1,2020-05-12,"Email_Campaign_Wikipedia_Challenge2020"
"4","?campaign=WMDE_2020_challenge_5","de.wikipedia.org/wiki/Spezial:Benutzerkonto_anlegen",1,2020-05-12,"Email_Campaign_Wikipedia_Challenge2020"
"5","?campaign=WMDE_2020_challenge_7","commons.wikimedia.org/wiki/Commons:Willkommen",2,2020-05-12,"Email_Campaign_Wikipedia_Challenge2020"

And everything observed on 2010/05/13 thus far:

"","Tag","Page","Pageviews","date","campaign"
"1","?campaign=WMDE_2020_challenge_12","de.wikipedia.org/wiki/Spezial:Anmelden",1,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"2","?campaign=WMDE_2020_challenge_14","de.wikipedia.org/wiki/Special:MyPage/Artikelwerkstatt",1,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"3","?campaign=WMDE_2020_challenge_14","de.wikipedia.org/wiki/Spezial:Anmelden",1,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"4","?campaign=WMDE_2020_challenge_16","de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement/Literatur",1,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"5","?campaign=WMDE_2020_challenge_18","de.wikipedia.org/wiki/Spezial:Anmelden",1,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"6","?campaign=WMDE_2020_challenge_24","de.wikipedia.org/wiki/Spezial:Anmelden",1,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"7","?campaign=WMDE_2020_challenge_25","de.wikipedia.org/wiki/Spezial:Anmelden",1,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"8","?campaign=WMDE_2020_challenge_26","de.wikivoyage.org/wiki/Hauptseite",2,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"9","?campaign=WMDE_2020_challenge_5","de.wikipedia.org/wiki/Spezial:Benutzerkonto_anlegen",2,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"
"10","?campaign=WMDE_2020_challenge_7","commons.wikimedia.org/wiki/Commons:Willkommen",4,2020-05-13,"Email_Campaign_Wikipedia_Challenge2020"

@Christine_Domgoergen_WMDE Please let me know if there is something that you can remember you have tested but that still cannot be found in these test results.
Also I am a little bit concerned about things like

www.wikidata.org/w/index.php?title=Q16943273&tour=wbitems&data=ok
commons.wikimedia.org/w/index.php?title=Special:UploadWizard&uselang=de

and similar dynamically generated content - if you remember, we've already asked that question: can they be tracked as everything else in the wmf.webrequest table (@Addshore: what is your take on this?)

@GoranSMilovanovic Okay, so the missing tag is no problem in detecting the page views, only in finding the user registrations?

Do you think we need to activate the patch in the other projects in order to track the user registrations and editing behavior?

I have for sure produced more page views during the evening but I cannot say for sure which ones and how often, sorry I didn't record it since it was for testing something else.

Special pages: yes I remember, but I don't remember if we found a solution.

@Christine_Domgoergen_WMDE Wrapping up:

• we need the patch deployed to other wikies beyond dewiki for this campaign so that the campaign tags enter the event.serversideaccountcreation table;
• we need to figure out if dynamically generated content (see examples in the bottom of T251524#6133949) are tracked in the wmf.webrequest table as they are.

@Christine_Domgoergen_WMDE

Okay, so the missing tag is no problem in detecting the page views, only in finding the user registrations?

Yes.

Do you think we need to activate the patch in the other projects in order to track the user registrations and editing behavior?

I would say yes, but again: that is a question for people who develop our logging and tracking procedures, not for me who is doing analytics.

I have for sure produced more page views during the evening but I cannot say for sure which ones and how often, sorry I didn't record it since it was for testing something else.
Special pages: yes I remember, but I don't remember if we found a solution.

I am not worried about static pages at all; can you remember if you have ever tested any of the following two:

www.wikidata.org/w/index.php?title=Q16943273&tour=wbitems&data=ok
commons.wikimedia.org/w/index.php?title=Special:UploadWizard&uselang=de

If you did not: (1) please do, (2) make sure that the campaign tag is attached when you test, and (3) please let me know when you did it.
I will then try to find the respective pageviews in the wmf.webrequest table. Thanks.

@GoranSMilovanovic
Yes, thank you!

we need the patch deployed to other wikies beyond dewiki for this campaign so that the campaign tags enter the event.serversideaccountcreation table;

Who could help us here, do you have an idea?

we need to figure out if dynamically generated content (see examples in the bottom of T251524#6133949) are tracked in the wmf.webrequest table as they are.

And who could help us with this?

@Christine_Domgoergen_WMDE

Who could help us here, do you have an idea?

The one and the ultimate hero of the Adam's patch: @Addshore

And who could help us with this?

I really do not know. If you produce the pageviews as I have suggested in T251524#6134154 I will be at least able to tell you if they end up in the wmf.webrequest table or not. What worries me is the following (a) it could be that dynamically generated content does not enter the webrequest table at all, or (b) it could be that dynamically generated content's URL is somehow transformed before entering the above mentioned table. And I have no answers to offer to either (a) or (b). But I guess: @Addshore

@Christine_Domgoergen_WMDE

Please let me know when the patch is deployed and the campaign has started.
In the meantime I really hope someone will help us figure out our dynamic content tracking dilemma.

@Addshore Could you help us here with deploying the patch? And if you have an idea about Goran's two questions about the dynamically generated content of special pages it would be very appreciated as well :-) Thank you!

Edit: we need the patch in Wikidata, Wikimedia Commons, Wikivoyage and Wiktionary

@GoranSMilovanovic The campaign started yesterday evening as planned. Apart from the problem with tracking page views and user registrations in projects other than dewiki and the dynamic pages the tracking is place and running, right?

If the patch is deployed later during the campaign can you track user registrations and page views only from the moment the patch is in place or does the patch enable you to track those even before it was there?

@Christine_Domgoergen_WMDE @WMDE-leszek

The campaign started yesterday evening as planned. Apart from the problem with tracking page views and user registrations in projects other than dewiki and the dynamic pages the tracking is place and running, right?

If you say the campaign has started, then I guess it started, right? In that case, I will (1) deploy the tracking procedures for this campaign today, with (2) a test run of the update procedure scheduled for late CET hours tonight (to give it some time to accumulate the data), and (3) start daily reporting in a Google Drive spreadsheet as of tomorrow.

If the patch is deployed later during the campaign can you track user registrations and page views only from the moment the patch is in place or does the patch enable you to track those even before it was there?

I would say the data will be available only once the patch is deployed across all the projects that we need to track. Otherwise, if I remember @Addshore's code correctly, it goes for dewiki only.

@GoranSMilovanovic Yes, thank you for setting up everything! We expect the first data on dewiki on day 4 of the campaign. The first data in one of the other projects is not expected until day 7, so if the patch is deployed unitl then we are good. I will be out of office next week but @Verena can help you if any questions come up.

In T251524#6136215, @Christine_Domgoergen_WMDE wrote:

@Addshore Could you help us here with deploying the patch? And if you have an idea about Goran's two questions about the dynamically generated content of special pages it would be very appreciated as well :-) Thank you!

Edit: we need the patch in Wikidata, Wikimedia Commons, Wikivoyage and Wiktionary

I certainly can deploy a change for it on Monday.
I would appreciate a ping then as I'm likely to miss it as it isn't in my regular working set any more!

It will probably be much easier to just deploy this cod wall sites nowz then we also won't run into future issues with it not working of further sites.

@Christine_Domgoergen_WMDE @Verena

T251524#6140399

We expect the first data on dewiki on day 4 of the campaign. The first data in one of the other projects is not expected until day 7, so if the patch is deployed unitl then we are good.

No problem at all. Day 4 of the campaign is today (May 17, 2020 - see campaign start date in the description of the ticket), the analytics code will be deployed tonight and it works independently of how many wikies are tracked.

@Addshore Could you give us an update if the patch was deployed today?

Hi sorry it was not!
This ticket totally slipped my mind.
It's possible that I will be able to do an alteration this evening.
It would be good to have a better process for things such as this.

Pings in mattermost work better than pings here (sorry I should have said that)

@Verena @Addshore

Pings in mattermost work better than pings here (sorry I should have said that)

That being said: what Mattermost channel does the New Editors team use?

Change 597336 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/WikimediaEvents@master] Run onBeforeInitializeWMDECampaign hook on all sites

https://gerrit.wikimedia.org/r/597336

@Addshore Given the discussion of this patch (c.f. @Krinkle comments), I need to ask: do we have a go for tracking and analytics here?

@Verena Daily reporting for this campaign with all the data collected thus far is found in this spreadsheet.

I could not fully understand the discussion on gerrit, but deploying the patch seems more complicated than expected. The possibility to track registrations would be nice but it is not a must have for this campaign.

If there is more work on it necessary than you can invest right now, we would be fine with setting the deployment on hold for the next campaign.

In T251524#6151523, @GoranSMilovanovic wrote:

@Verena Daily reporting for this campaign with all the data collected thus far is found in this spreadsheet.

Thank you!

In T251524#6149010, @GoranSMilovanovic wrote:

@Verena @Addshore

Pings in mattermost work better than pings here (sorry I should have said that)

That being said: what Mattermost channel does the New Editors team use?

Until now we didn't have a specific channel but will consider to set up one now.

In T251524#6151795, @Verena wrote:

I could not fully understand the discussion on gerrit, but deploying the patch seems more complicated than expected. The possibility to track registrations would be nice but it is not a must have for this campaign.

If there is more work on it necessary than you can invest right now, we would be fine with setting the deployment on hold for the next campaign.

Sounds good to me.
I guess you need to coordinate with @tobi_wmde_swe to get dev time for this in advance of needing it.

@Addshore Hi Adam, thank you for your help!

Sorry, I am not sure from the comments: has the patch been deployed now and can we track in the other projects or not?

@GoranSMilovanovic

@Christine_Domgoergen_WMDE

Please see T251524#6151795 @Verena:

If there is more work on it necessary than you can invest right now, we would be fine with setting the deployment on hold for the next campaign.

and T251524#6153434 @Addshore:

Sounds good to me. I guess you need to coordinate with @tobi_wmde_swe to get dev time for this in advance of needing it.

@GoranSMilovanovic Okay, thank you for the clarification!

Data acquisition completed:

• fetching user edits now
• working on the preliminary campaign report.

@Christine_Domgoergen_WMDE Here is the campaign report for this one:

@GoranSMilovanovic Thank you! I will get back to you about this one next week.

@GoranSMilovanovic Hi Goran, this ticket is is done, no further questions :-) I already published the report and will close the ticket. Thank you for your work on this!

Change 597336 abandoned by Addshore:
[mediawiki/extensions/WikimediaEvents@master] Run onBeforeInitializeWMDECampaign hook on all sites

Reason:

https://gerrit.wikimedia.org/r/597336