@Christine_Domgoergen_WMDE @GoranSMilovanovic

2020/09/15 1:1 notes:

• the first stage: banner -> landing page 1 (not in Wikipedia)
  • Banner impressions will be tracked from the wmf.webrequest table;
  • Banner clicks and closing clicks will be tracked from the event.WMDEBannerInteractions - to be checked if this schema is reliable @kai.nissen
• the second stage is standard (everything is on-Wiki and tagged from the e-mails)
• the e-mails, campaign tags, and landing pages for the second stage are found in the Tracking concept - Banner campaign: 30 Days Wikipedia Challenge document.

@kai.nissen Hey Kai, we will be tracking the banner data from two sources now, the banner impressions from the wmf.webrequest table and the banner clicks and closing clicks fom the event.WMDEBannerInteractions schema. We were wondering how reliable the data from the schema will be this time (targeting only non-logged in users, no targeting in the browser). Can we have your opinion on this? Thank you!

Banner clicks may still be unreliable, because clicking loads a new page. Closing clicks and banner impressions should be fine, though.

@kai.nissen Thank you.

@GoranSMilovanovic So Goran, what do you think? We have to track banner clicks and closing clicks via the schema, because the landing page is offwiki and we cannot track closing clicks via wmf.webrequest. What do you prefer in regard to banner impressions, tracking them via the schema or via wmf.webrequest? It might be tidier to have all data from one source, maybe?

@Christine_Domgoergen_WMDE Please see: https://phabricator.wikimedia.org/T249617#6478906 which says:

(B) rely on the assumption that its seen_by field reports the banner impressions data correctly while its clicked_by field reports incorrect data.

This means that we will use the event.WMDEBannerActions schema and its seen_by field for for banner impressions, while the banner clicks will be inferred from pageviews relying on the assumption that each banner click necessarily produces a pageview.
Please confirm. Thank you.

Now, as of:

It might be tidier to have all data from one source, maybe?

That is not of essential importance as long as our data prove to be correct and consistent, but yes, typically people tend to keep track of consistent data from a constrained set of consistent resources : )

We have to track banner clicks and closing clicks via the schema, because the landing page is offwiki and we cannot track closing clicks via wmf.webrequest.

As far as I understand from https://phabricator.wikimedia.org/T249617 the banner clicks are not recorded correctly in event.WMDEBannerActions schema and we will use pageviews instead: a banner click == a pageview.

What do you prefer in regard to banner impressions, tracking them via the schema or via wmf.webrequest?

As far as I understand from https://phabricator.wikimedia.org/T249617 the banner impressions will be taken from the seen_by field of the event.WMDEBannerActions schema.

@kai.nissen @Christine_Domgoergen_WMDE Are these the test banners for this campaign: T262670#6474604?

@GoranSMilovanovic

As far as I understand from https://phabricator.wikimedia.org/T249617 the banner clicks are not recorded correctly in event.WMDEBannerActions schema and we will use pageviews instead: a banner click == a pageview.

Yes that would be ideal, BUT: the landingpage is offwiki, so as I understand we cannot track page views from the wmf.webrequest table but have to rely on the banner clicks counted in the schema, even if it might be incorrect, this is our only possibility to track those. Correct?

As far as I understand from https://phabricator.wikimedia.org/T249617 the banner impressions will be taken from the seen_by field of the event.WMDEBannerActions schema.

Okay, perfect!

@Christine_Domgoergen_WMDE

Yes that would be ideal, BUT: the landingpage is offwiki, so as I understand we cannot track page views from the wmf.webrequest table but have to rely on the banner clicks counted in the schema, even if it might be incorrect, this is our only possibility to track those. Correct?

I do not think so. Why use a resource for which we already know that it used to be unreliable? @WMDE-leszek @kai.nissen Matomo, if possible, and I also believe we have used Piwik in the past?

@GoranSMilovanovic Yes we will track the page views via Matomo also. I would like to verify how unreliable it really is when we do not have a complicated tracking of the target groups and it really is no extra effort to use all the data from the schema, right?

In T262534#6478963, @GoranSMilovanovic wrote:

@kai.nissen @Christine_Domgoergen_WMDE Are these the test banners for this campaign: T262670#6474604?

@kai.nissen Can we use the banners from https://phabricator.wikimedia.org/T262670 for the tracking test already?

@Christine_Domgoergen_WMDE The banners are up. You can do tracking tests using these links:

Control banner

• desktop
• ipad
• mobile

Variant banner

• desktop
• ipad
• mobile

@Christine_Domgoergen_WMDE I am about to test the campaign banners now.

@Christine_Domgoergen_WMDE @kai.nissen

Well, nothing from the banners is currently found under /beacon/impression in the wmf.webrequest table, but I guess they were just recently put online so we will have to wait some time for the table to pick them up.

Continuing tests + preparations for daily reporting.

@Christine_Domgoergen_WMDE @kai.nissen Also, nothing is still found in event.WMDEBannerInteractions on these banners; testing postponed, we will wait for a few hours.

@Christine_Domgoergen_WMDE As of T262534#6483268: from what I see the landing page is on https://de.wikipedia.org/ - it is the main page, in fact - so no offwiki pages in this campaign after all?

@Christine_Domgoergen_WMDE Disregard T262534#6501396, it's a page on https://www.wikimedia.de/ not wikipedia, Ok... so we will need the pageviews for the landing page measured in an alternative way - let me know which one will you choose and where will the data be served. Thanks.

@Christine_Domgoergen_WMDE @kai.nissen @WMDE-leszek A summary:

• banner impressions (from wmf.webrequest, not event.WMDEBannerInteractions schema) code is in place; nothing yet on the campaign banners is found;
• banner interactions (from event.WMDEBannerInteractions schema) code is in place; to be used for data on closing clicks only;
• pageviews code is in place; pageviews are assumed to deterministically generated by the respective banner clicks in this campaign;
• pageviews data collection for the fist landing page is not set: you need to let me know where from will I be getting the data for this;
• user registrations code is in place;
• user edits code is in place.

@GoranSMilovanovic Thank you for the summary! Please see my comments below:

banner impressions (from wmf.webrequest, not event.WMDEBannerInteractions schema) code is in place; nothing yet on the campaign banners is found;

I assume the banner impressions will only be in the table once the campaign is up and running. Correct? Or is there any possibility to test this before?

banner interactions (from event.WMDEBannerInteractions schema) code is in place; to be used for data on closing clicks only;

Great. Please use the schema also for the data on banner clicks.

pageviews code is in place; pageviews are assumed to deterministically generated by the respective banner clicks in this campaign;
pageviews data collection for the fist landing page is not set: you need to let me know where from will I be getting the data for this;

Since the landingpage is offwiki (https://www.wikimedia.de/wikipedia-challenge/) we cannot track pageview from the wmf.webrequest table. We will use the schema to track the banner clicks instead (even though it might be unreliable)

user registrations code is in place;
user edits code is in place.

Great. This refers to the second part of the tracking: the user registrations and edits coming from the emails and the respective campaign tags in the emails, right?

I assume the banner impressions will only be in the table once the campaign is up and running. Correct? Or is there any possibility to test this before?

In the previous campaigns we were able to test the banners before actually running the campaign, or at least that is what I remember.

Great. Please use the schema also for the data on banner clicks.

Are you sure about this? Because in the previous campaign we have faced a problem with incompatible banner clicks and pageviews.

Since the landingpage is offwiki (https://www.wikimedia.de/wikipedia-challenge/) we cannot track pageview from the wmf.webrequest table. We will use the schema to track the banner clicks instead (even though it might be unreliable)

Does that mean that the pageviews for the landing page 1 will not be tracked in any way?

Great. This refers to the second part of the tracking: the user registrations and edits coming from the emails and the respective campaign tags in the emails, right?

Exactly.

@GoranSMilovanovic

In the previous campaigns we were able to test the banners before actually running the campaign, or at least that is what I remember.

@kai.nissen Can we test with the dummy banners if the tracking of banner impressions and banner clicks via the schema works before the campaign starts?

Are you sure about this? Because in the previous campaign we have faced a problem with incompatible banner clicks and pageviews.

Yes, but this is our only option. So yes, I am sure :-)

Does that mean that the pageviews for the landing page 1 will not be tracked in any way?

No. They will be tracked via matomo.

@GoranSMilovanovic
So I clicked twice on each banner from the links above, and produced one banner click and one closing click each. Can you find data now?

@GoranSMilovanovic
Just to be safe and to test the second part of the tracking as well I just created a user account from email 1 (campaign tag: ?campaign=WMDE_2020_challenge_1) with the user name "Testuser Challengebnr". Can you check, if you find it in the database? Thank you!

@Christine_Domgoergen_WMDE @kai.nissen

The banner impressions test (T262534#6502096), testing exactly the following banners:

• WMDE_challengebnr_fall2020_ctrl
• WMDE_challengebnr_fall2020_ctrl_ipad
• WMDE_challengebnr_fall2020_ctrl_mobile
• WMDE_challengebnr_fall2020_var
• WMDE_challengebnr_fall2020_var_ipad
• WMDE_challengebnr_fall2020_var_mobile

Results:

(A) From the /beacon/impression uri_path field, de.wikipedia.org and de.m.wikipedia.org uri_host field, of the wmf.webrequest table: nothing found.
(B) From the event.WMDEBannerInteractions table: found 85 data points (I've been clicking the banners too, yesterday), everything seems to be in place, all six banners are present in the data.

Proceeding to user registration tests now.

In T262534#6504826, @GoranSMilovanovic wrote:

(A) From the /beacon/impression uri_path field, de.wikipedia.org and de.m.wikipedia.org uri_host field, of the wmf.webrequest table: nothing found.

This is to be expected. If the banner is displayed using the banner parameter, impressions are not being logged.

@Christine_Domgoergen_WMDE @kai.nissen

User registration test (T262534#6504176) completed, found it:

year    month   day     hour    campaign                userid  username                webhost                 wiki
2020    9       30      7       WMDE_2020_challenge_1   3574641 Testuser Challengebnr   de.wikipedia.org        dewiki

@kai.nissen Thanks for T262534#6504832

@GoranSMilovanovic Great, thank you! So everything works and we are good to go :-) @kai.nissen

@Christine_Domgoergen_WMDE

The daily reporting procedures for this campaign are on and I am running the first daily update right now.

Because of the changes in the selection of wikis/pages for this campaign, I would like to check additionally the list of pages that we are tracking for pageviews.

From the campaign tracking document, these are the only pages that remain to be tracked:

https://de.wikipedia.org/wiki/(274301)_Wikipedia?tour=einfuhrung?campaign=WMDE_2020_challenge_4

https://de.wikipedia.org/wiki/Spezial:Benutzerkonto_anlegen?campaign=WMDE_2020_challenge_5

https://de.wikipedia.org/wiki/Spezial:Anmelden?campaign=WMDE_2020_challenge_12

https://de.wikipedia.org/wiki/Special:MyPage/Artikelwerkstatt?campaign=WMDE_2020_challenge_14

https://de.wikipedia.org/wiki/Benutzer:Trainingskonto_(WMDE)/Wikipedia_Training

https://de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement/Literatur

Please check out the list once again, just in case, and let me know if there are any additional pages that we need to track. Thank you.

The daily update will be run with the list of pages given here, but it can be easily corrected if there turns out that there are more pages to track.

@Christine_Domgoergen_WMDE Please do not forget to let me know where will the data on the pageviews for the first landing page (the offwiki one) be stored. Thank you.

@Christine_Domgoergen_WMDE @kai.nissen @WMDE-leszek

Here is the campaign daily report: spreadsheet.

Please note the following:

• the seen_by field in the Banner Interactions tab where the data from event.WMDEBannerInteractions are present is obviously showing incorrect numbers (e.g. compare the number of seen_by observations with the clicked_by and closed_by fields);

• in the Banner Interactions tab we find more campaign banners than in the Banner Impressions tab (data from wmf.webrequest); please check your campaign tracking definitions and let me precisely what banners do you want me to report on. The banners were previously defined in T262534#6501120. Thank you.

@GoranSMilovanovic Hi, Goran, thank you for the update!

Please check out the list once again, just in case, and let me know if there are any additional pages that we need to track. Thank you.

Okay, I will have a look again, but I think we are good here.

in the Banner Interactions tab we find more campaign banners than in the Banner Impressions tab (data from wmf.webrequest); please check your campaign tracking definitions and let me precisely what banners do you want me to report on. The banners were previously defined in T262534#6501120. Thank you.

I have no idea, where the additional banners come frome. @kai.nissen can you help here?

During the last campaign we used a banner name builder that suffixed the banner name by "ipad" or "mobile". This time, we are using Central Notice's device detection and have hard coded device type suffixes. But the banner name builder is still suffixing the keywords.

@GoranSMilovanovic A fix for this is ready, do you want us to use new keywords to be able to distinguish?

What the suffixes on the interactions sheet most likely mean:

@kai.nissen Would that action result in six distinguishable banners as they were previously described in this ticket and as they are now found in the Banner Impressions tab (the wmf.webrequest data) of the campaign daily reporting spreadsheet?

@GoranSMilovanovic Yes, the fix does exactly that.

@kai.nissen Please apply the fix then.

@kai.nissen @Christine_Domgoergen_WMDE @WMDE-leszek

We are still getting 14 different banners from the event.WMDEBannerInteractions schema, in contrast with six banners defined in the scope of this campaign.

@Christine_Domgoergen_WMDE @kai.nissen @WMDE-leszek

As of the list of the pages that need to be tracked in this campaign - see T262534#6511503 - please take into your consideration the following:

• we have two user registrations observed, on 2nd and 3rd of October 2020, in the campaign spreadsheet, while
• no pageviews observed on 2nd and 3rd of October 2020.

Once again: are we sure that we are tracking all the pages that we need to track?

@Christine_Domgoergen_WMDE @kai.nissen @WMDE-leszek

Please note that we are still not getting any pageviews data on the campaign pages as listed in T262534#6511503, while we are still getting user registrations from this campaign: daily report.

@Christine_Domgoergen_WMDE @kai.nissen @WMDE-leszek

The daily update is ready: we are still receiving user registrations while not observing any pageviews. Please check your campaign settings and advise because I am using the same code to track the pageviews as it was used in our previous campaigns - so I guess the problem is not in the analytics code. However, I think we better have everything checked. Thank you.

@GoranSMilovanovic Yes, no worries, we do not need the pageviews of all sub-landing pages because there are so many. We focus on some of them and decided not to track others. So it is possible that you get user registrations from other sub-landing pages.

@Christine_Domgoergen_WMDE @kai.nissen Congrats, 38 user registrations from this campaign thus far... daily update.

@Christine_Domgoergen_WMDE @kai.nissen 87 registrations from this campaign thus far: daily update.

@Christine_Domgoergen_WMDE @kai.nissen @WMDE-leszek

The daily reporting for this campaign is completed: spreadsheet.

For the purposes of campaign analytics (Preliminary report), the following 14 banners observed from the event.wmdebannerinteractions table:

1. WMDE_challengebnr_fall2020_ctrl
2. WMDE_challengebnr_fall2020_ctrl_ipad
3. WMDE_challengebnr_fall2020_ctrl_ipad_ipad
4. WMDE_challengebnr_fall2020_ctrl_ipad_mobile
5. WMDE_challengebnr_fall2020_ctrl_mobile
6. WMDE_challengebnr_fall2020_ctrl_mobile_ipad
7. WMDE_challengebnr_fall2020_ctrl_mobile_mobile
8. WMDE_challengebnr_fall2020_var
9. WMDE_challengebnr_fall2020_var_ipad
10. WMDE_challengebnr_fall2020_var_ipad_ipad
11. WMDE_challengebnr_fall2020_var_ipad_mobile
12. WMDE_challengebnr_fall2020_var_mobile
13. WMDE_challengebnr_fall2020_var_mobile_ipad
14. WMDE_challengebnr_fall2020_var_mobile_mobile

will be reduced to the following six banners defined in this campaign:

1. WMDE_challengebnr_fall2020_ctrl
2. WMDE_challengebnr_fall2020_ctrl_ipad
3. WMDE_challengebnr_fall2020_ctrl_mobile
4. WMDE_challengebnr_fall2020_var
5. WMDE_challengebnr_fall2020_var_ipad
6. WMDE_challengebnr_fall2020_var_mobile

by using the first two banner suffixes and neglecting the third banner suffix where present in the set of 14 banners from event.wmdebannerinteractions.

@GoranSMilovanovic Thank you. The reporting of user registrations needs to be continued till November 11 because users receive emails until then. Could you update the spreadsheet? Thank you!

@Christine_Domgoergen_WMDE Done: campaign spreadsheet.

great, this looks really good :-) thank you!

@Christine_Domgoergen_WMDE The campaign preliminary report, including all data from banner impressions, banner interactions, pageviews, user registrations, and user edits available until now:

Note: I still did not receive any data on the pageviews for LP1 from Matomo.

@GoranSMilovanovic great, thank you. No we track the pageviews from LP1 via Matomo separately and I don't think its necessary to include them in your report.

@Christine_Domgoergen_WMDE @WMDE-leszek

Here is an update of the Campaign Report from the most recent user registrations and edits data.

Please find some time to review the report and let me know if there is anything that needs to be corrected or changed, as well as if there are any sections or analyses that you would like to be included there.

This campaign will be tracked until 2021. It will be difficult and time consuming if any changes will need to be addressed only then, because I guess we will have more campaigns in the meantime and I also need to focus on my Wikidata related work. So we should not leave this campaign out of sight until next year.

Thank you.

@GoranSMilovanovic Thank you! I will have a look next week and get back to you. For your planning: As far as I know there are no other campaigns planned from our side for the rest of the year. At the beginning of next year, we probably will need your expertise again though ;-)

@Christine_Domgoergen_WMDE Thank you for helping me plan my work!

@Christine_Domgoergen_WMDE Fresh data:

Hi @GoranSMilovanovic, I had a look at the report and it would be great if you could include the following:

• Section 0 Banner impressions per device: Can you include a chart similar to 0.1.2. which summarizes impressions, clicks and closing clicks per banner per device?
• Section 0 Banner impressions var/ctrl: Can you include a chart similar to 0.1.2. which summarizes impressions, clicks and closing clicks per banner A/B?
• Section 2 User Registrations: Can you include the total number of user registrations (all channels, all days) somewhere?
• Section 3.3 User Edits per Campaign Channels: where does the number of edits without associated channel (NA) come from? How can you tell they edited via the campaign, if the there is no campaign tag associated?
• Section 3.4. User Edits: additionally to the total number of edits it would be interesting for us to see, when the users edit: do they just edit shortly after the campaign or do the continue editing afterwards for a longer period of time? Could you include a chart which shows this?
• Section 3.4. User Edits in Edit Classes: the first row of the table (0-1 edits) is really 1 edit, correct? We do not include the users registered but having 0 edits here. Could you adjust the caption accordingly?

Thank you!

@Christine_Domgoergen_WMDE

Section 0 Banner impressions per device: Can you include a chart similar to 0.1.2. which summarizes impressions, clicks and closing clicks per banner per device?

See: Chart 0.1.6B Impressions, clicks and closing clicks per banner , per device.

Section 0 Banner impressions var/ctrl: Can you include a chart similar to 0.1.2. which summarizes impressions, clicks and closing clicks per banner A/B?

See: Chart 0.1.7B Mean, median, and total Impressions, clicks and closing clicks per banner.

Section 2 User Registrations: Can you include the total number of user registrations (all channels, all days) somewhere?

See Section 2.1 User Registrations Overview, immediately bellow the chart.

Section 3.3 User Edits per Campaign Channels: where does the number of edits without associated channel (NA) come from? How can you tell they edited via the campaign, if the there is no campaign tag associated?

There was no campaign tag associated with those users and they were removed from the analysis (not too many of them, one only I think).

Section 3.4. User Edits in Edit Classes: the first row of the table (0-1 edits) is really 1 edit, correct? We do not include the users registered but having 0 edits here. Could you adjust the caption accordingly?

Done. Additionally, the sections 3.4.* (edit classes per campaign tag) are now consolidated in one single chart.

Section 3.4. User Edits: additionally to the total number of edits it would be interesting for us to see, when the users edit: do they just edit shortly after the campaign or do the continue editing afterwards for a longer period of time? Could you include a chart which shows this?

I think that the chart in sections 3.1 User Edits Overview and 3.2 User Edits by Campaign Channels show exactly what you need?

Here is an updated report:

@GoranSMilovanovic Great, thank you! It will be really interesting how the editing numbers develop until the end of the year....

@Christine_Domgoergen_WMDE Update as of 2020/11/15 22:30 CET

@Christine_Domgoergen_WMDE Report as of 2020/11/24 14:00 CET

Note: I had to introduce an additional edit class > 100 edits - because of one user who has already made > 100 of them. Congrats.

@Christine_Domgoergen_WMDE Report as of 2020/11/30

@GoranSMilovanovic Great, thank you.

I have one question: when I add on the numbers of registrations via campaign tag (2.3) I get 346 and not 353 as in 2.1. Can you check again? Thank you!

@Christine_Domgoergen_WMDE I am on it.

@Christine_Domgoergen_WMDE Found it:

• that is because there were some users who have registered after 2020-11-11 - exactly seven (7) of them, and
• they are being taken into account in 2.1. (the part which says The total number of users registered in this campaign is 353),
• but not in 2.3.

From the ticket description:

Last User finishing mailing campaign: November 11th

How does it happen that we still observe registrations following November 11th?

I have updated the report (for edits data) and consolidated it not to count in anyone who has registered following November 11th, so now there should be no mismatch between 2.1 and 2.3 anymore.

@GoranSMilovanovic Ah perfect, thank you! I guess they might have started the Challenge a bit later maybe or decided later to register while the cookie was still in place. Although we stated the 11.11. as the end of the campaign I think it would be good to include them also. This would be the last change of the report I hope! :-)

Thank you and have a nice Christmas time!

@Christine_Domgoergen_WMDE Done.

@GoranSMilovanovic Great, thank you! The intro says the last update of user edit data is from November 24th, is that true? Because for the tracking of six week retention we would need the editing behavior until December 23rd as stated in the task description.

Thanks for checking and have good start in 2021!

@Christine_Domgoergen_WMDE Exactly, sorry about that. I will updated it very soon and share the final report here with you.

@Christine_Domgoergen_WMDE

The intro says the last update of user edit data is from November 24th, is that true?

No, it is not: the observation window for this campaign ends on December 24th as requested, and the datasets are aligned with that.
Please find the report with the corrected Intro here:

I have double-checked everything by re-running the edits data gathering procedures, and all seems fine except for one thing which is completely puzzling:

• In the report in T262534#6711614 we observe one user edit on December 23rd, 2020;
• In the current update - the one provided here - that one user edit is missing (with the last observed edit occurring on December 21st, 2020, which fits Ok because there were not edits on December 22nd, 2020).

Nothing has changed in our data gathering procedures - we are using exactly the same code as we have used during the campaign and afterwards. So it must be something in the databases - and that is, unfortunately, out of my reach.

I also hope your 2021 is going fine so far.

@GoranSMilovanovic Ah perfect, thanks for checking and updating! Thanks also for double checking the figures. The missing edit indeed sounds weird... but I guess we'll have to work what the databases are giving us ;-)

I will close the ticket now, thank you again for all your work and help! I'll be in touch about our next banner campaign towards the end of January. It is planned to run in mid March and there will also some other tracking be involved, just for your planning.

Have a good rest of the week!