Page MenuHomePhabricator

Create daily tracking reports for campaign: Banner campaign for occasional editors 2020
Closed, ResolvedPublic

Description

Please provide daily reports during the campaign (14th May until 27th May) and afterwards weekly reports on edit numbers until July 7th.

There will be one campaign with 9 different banners (including mobile versions) which target already logged in users. Campaign goal are edits (not registrations). Further information on user journeys, landing page links and campaign tags in this document.

campaign information
central notice settings
duration: 14th May until 27th May
note: The campaign will run parallely to this year's WikiLovesEarth-Campaign but it won't affect analytics.

target groups
(targeting for banner display will be executed by API request of the banner itself)
Target group A: eligible voters
more than 200 edits in total
less than 1000 edits in total
registered for at least two months
last edit in 2019 or 2020

Target group B: new users
20 or more edits in total
less than 200 edits in total

The a/b test scope is comparing target group specific banners (1 and 2) against banner 3:
banner 1: target group A,
banner 2: target group B,
banner 3: target group A+B,

Tracking is dependent on eventlogging. The required event scheme will be updated https://phabricator.wikimedia.org/T250791
and needs to be tested before campaign start.

daily reporting during campaign
The following information should be included in the daily reporting

  • impressions per banner
  • clicks per banner/ page views per landing page
  • closing rate of banners

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Verena updated the task description. (Show Details)Apr 7 2020, 1:51 PM
Verena updated the task description. (Show Details)Apr 30 2020, 3:02 PM

@GoranSMilovanovic : We made some progress here. The eventlogging scheme will be updated so that user ids should be available. Testing of that is going to happen next week (probably)

Verena updated the task description. (Show Details)May 6 2020, 3:05 PM
Verena added a subscriber: Christine_Domgoergen_WMDE.
Verena added a comment.May 6 2020, 3:39 PM

Update: Deployment of scheme update will be on Monday, 11th May.

Verena updated the task description. (Show Details)May 6 2020, 3:47 PM
GoranSMilovanovic renamed this task from Create daily tracking reports for campaign to Create daily tracking reports for campaign: Banner campaign for occasional editors 2020.May 7 2020, 11:08 AM

@Verena @Christine_Domgoergen_WMDE Wrapping up:

  • no pageviews with the campaign tags in the uri_query field of wmf.webrequest found: T251535#6134075
  • the event.WMDEBannerInteractions table delivers nicely.
GoranSMilovanovic added a comment.EditedMay 13 2020, 8:47 PM

@Verena @Christine_Domgoergen_WMDE

Has this campaign started - beyond testing - on May 13, as this tickets says it should have?

Because... no banner impressions are registered on 2020/05/13:

SELECT uri_query FROM wmf.webrequest WHERE (uri_host = 'de.wikipedia.org' OR uri_host = 'de.m.wikipedia.org') AND uri_path = '/beacon/impression' AND uri_query LIKE '%WMDE_oceditors_spring_2020_%' AND year = 2020 AND month = 5 AND day = 13;

results in:

uri_query
No rows selected (160.434 seconds)

Also, from event.centralnoticeimpression:

select dt, wiki, event.device, event.campaign, event.banner, event.bucket, event.banner_count from event.centralnoticeimpression where year=2020 and month=5 and day=13 and event.banner like "%WMDE_oceditors_spring_2020_%"

results in:

dt      wiki    device  campaign        banner  bucket  banner_count
No rows selected (66.303 seconds)

@GoranSMilovanovic No, due to the testing and some issues with the code we decided to start today

@Christine_Domgoergen_WMDE Ok - then we will also start the analytics daily cycle today.

GoranSMilovanovic added a subscriber: kai.nissen.EditedMay 15 2020, 10:15 AM

@Christine_Domgoergen_WMDE @Verena @kai.nissen

The daily reporting for this campaign has started. The campaign spreadsheet is here.

Spreadsheet Guide

Banner Interactions Tab: data from the event.wmdebannerinteractions table

  • bannername - banner name
  • seen_by - the number of unique users who have seen the banner on a particular day
  • closed_by- the number of users who have closed the banner on a particular day
  • mean_close_imp - the mean number of impressions per user before they close the banner on a particular day
  • clicked_by - the number of users who have clicked the banner on a particular day
  • mean_click_imp - the mean number of impressions per user before they click the banner on a particular day
  • close_rate - closed_by divided by seen_by
  • click_rate - clicked_by divided by seen_by
  • day - day
  • campaign - this campaign

Banner Impressions Tab: data on banner impressions from the wmf.webrequest table and the /beacon/impression path

  • banner - banner name
  • device - device
  • impressions - how many banner impressions were served, adjusted for the recordImpressionSampleRate value as found in the respective record
  • date - date
  • campaign - this campaign

Pageviews Tab: data on pageviews from the wmf.webrequest table

  • Tag: the campaign tag as found in the uri_query field
  • Page: the target pages
  • Pageviews: number of pageviews on a particular date
  • date: date
  • campaign: this campaign

The spreadsheet will be updated daily until until 27th May. The update procedures run on stat1004 from crontab at 08:00 UTC each day and search for data from the previous day.

Please note that the banner impression data on the 2nd tab reflects the (approximate) number of times a banner was sent to users. Since the banner is only displayed to users that are in one of the target groups, the actual display number is lower.

@kai.nissen So that explains the discrepancy in the numbers that I have observed while comparing event.wmdebannerinteractions and wmf.webrequest. Thank you!

@Verena

So far we have observed only pageviews for de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement (see the campaign spreadsheet).

Please confirm that (from the campaign tracking concept):

Landing Page 2b: https://de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement/Literatur

Landing Page 2c: https://de.wikipedia.org/wiki/Wikipedia:Mentorenprogramm

are indeed tracked separately with appropriate tags implemented correctly in the banners.

Thank you.

@Verena @kai.nissen @WMDE-leszek Could someone please take a look at my question on pageviews in T249617#6142887 ^^
Thank you.

Also, @Verena in the campaign tracking doc, section Analytics, after campaign:

detailed research questions to be defined further during campaign...

please make sure that we have all the research questions formulated as soon as possible - the campaigns ends on Wednesday, May 27th, giving us only three days (May 27 - 29) to wrap-up the results because I will not be available in the following, first week of June 2020. Thus, I would prefer to have all the analytics code in place well before May 27th.

@GoranSMilovanovic
All users see the same [[ de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement | landing page ]] after their banner click. Landing Page 2b and 2c both are links on that page. These links don't contain tags. Would you need that to find the event in the database? Or is it possible to find out if users who clicked on the banners later also viewed one of these 2 pages or both?

@GoranSMilovanovic As for the research questions: We will do your best to have the questions ready but the analysis can also be done after your absence.

@Verena

T249617#6145202

All users see the same de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement after their banner click. Landing Page 2b and 2c both are links on that page. These links don't contain tags. Would you need that to find the event in the database? Or is it possible to find out if users who clicked on the banners later also viewed one of these 2 pages or both?

We need the tags on all pages that we need to track. We can determine who visited DeinEngagement first and then visited Landing Page 2b or 2c by looking at the referer field in the wmf.webrequest table, but we will not be able to separate who did that as a part of a campaign user journey from who did that spontaneously unless both Landing Page 2b and 2c are also tagged.

T249617#6145219

As for the research questions: We will do your best to have the questions ready but the analysis can also be done after your absence.

Thank you!

Ok, so where would we need these tags? I assume we have to add them to the links to landing page 2b and 2c which are on the first landing page?

@Verena

I assume we have to add them to the links to landing page 2b and 2c which are on the first landing page?

Well I guess yes, but you need to ask people who actually deploy those tags to make sure.

Because the campaign is already active and because cryptic tags in links on Wiki pages are usually considered as bad style we will have to refrain from adding these tags and accept some ambiguity concerning the page views von 2b and 2c.

@Verena

Because the campaign is already active and because cryptic tags in links on Wiki pages are usually considered as bad style we will have to refrain from adding these tags and accept some ambiguity concerning the page views von 2b and 2c.

  • Refactoring the analytics code now to include the pageviews from landing pages 2b and 2c that were produced from the DeinEngagement landing page irrespective of the campaign tags;
  • re-running daily pageviews update to account for landing pages 2b and 2c.

@Verena Now we have all the campaign landing pages listed in the Pageviews tab in the campaign spreadsheet. Note: we can recognize the campaign tags for the Landing Page 2b and 2c correctly and unambiguously, because the users carry their campaign tag from DeinEngagement to the respective referer field in wmf.webrequest for any pageview. I have previously overlooked that lucky circumstance for this campaign.

Thank you for the update and a relief that we have access to that information.

GoranSMilovanovic added a comment.EditedJun 12 2020, 8:17 PM

@Verena

Here is the report for this campaign:

The Report currently encompasses:

  1. Banner Interactions (click rates, closing rates, impressions)
  1. Campaign Pageviews
  1. User Edits: during the campaign vs. after the campaign, split by new tasks/community support and edit classes.

What is currently missing from the Report in respect to the Campaign Tracking Concept:

Analysis of target group editing behaviour before and after banner click
If possible: Split up in groups compared to their editing behavior before the campaign (see question 3), e.g. xx users edit xx percent more than before the campaign

Q1. How much time before and how much time after they click?

Set up cohorts (account age and editing behaviour regarding activity in the defined time frame)

ToDo once Q1. is answered.

Also, A/B tests were mentioned:

There will be A/B Testing with Banner 1/3 and Banner 2/3.

but Banner 3 - Thank you and find support + tasks (Banner 1 + 2 combined) - does not seem to have been implemented in the campaign. So we can do nt (new tasks) vs. cs (community support) A/B tests, but we need to decide on the criteria (A/B test on what: clicks, edits, or both).

@GoranSMilovanovic
Perfect, thank you! I will have a look at it and get back to you if there are questions.

@GoranSMilovanovic Hi Goran, I had a look at the report and your questions and for a better overview I added our research questions here in the tracking document. Please indicate if something is not clear! A lot of them are already answered in the report.

As I see a lot of charts have the daily numbers already so it would be great to also the totals (page views, clicks, impressions etc.) btw. the median (click rate, closing rate) for the data from the charts in 1, 2 and 3, if that is possible.

For the user edits, can you also calculate the average number of edits of users per day before, during and after the campaign?
Also it would be interesting to know if the number of users per edit class changed before, during and after the campaign. Could you include that in 3.4.?

I hope I didn't miss anything, please let me know if there are any questions.

@GoranSMilovanovic Oh I forgot one question: the campaign spreadsheet includes the numbers until 27.05.2020, the campaign ended on 28.5.2020 12.00 UTC though. Could you include the last day as well? Thank you!

@Christine_Domgoergen_WMDE Here is the updated version of the Campaign Report - I think it answers all of your questions:

Note. No Bayesian A/B tests were performed: from my previous experience in working with your team, sometimes numbers prima facie are enough, sometimes you ask for statistical A/B testing. Please let me know if you need statistical A/B testing included.

@GoranSMilovanovic Sorry for the late reply and thank you! I believe this is one of the best reports we ever had :-) And some really interesting results in the edit behavior! Still, while writing the summary some questions have come up:

  1. Table 1.1.2. could you give us three more decimal places here? Thank you.
  2. The numbers click rate mean + median and close rate mean and median from table 1.1.2 are really interesting for us. Could you also calculate them for the devices and add them to table 1.2.2 as well as for Banner Actions var vs. ctrl and add them to table 1.3.2. as well as for nt vs. cs and add them to table 1.4.2?
  3. Page Views per banner. I have a general question here: We had two distinctions: one into target groups (cs vs nt) and one into banners (ctrl vs. var). Since both target groups have seen banner var (non-variable text) we should be able to compare page views var vs. cs vs. nt - right? So I expect the page views of var + cs + nt to sum up to the total page views. However, when I sum up the page views of LP 1 nt=2089 + cs=1354 I already get the total page view number. Did I miss something? Is it possible to compare the page views var (both target groups) against ctrl.cs and ctrl.nt? Content-wise we had those three banners and we would like to know which of our three banners performed best.
  4. Edit activity: which group of users did you use for the analysis of the edit activity: those who saw the banner or those who clicked on the banner? How many unique users are those?
  5. Edit classes: to which timeframe do the edit classes refer to? Is it for example 2-4 edits in total (before campaign + during campaign + after campaign) or is it per day/wek/month in 3.4.1 - 3.4.2B?
  6. Edit classes 3.4.3 I summed up the number of users in the edit classes before, during and after the campaign and got a different number of users for each of the time spans: before 535, during 489 and after 427. Possible minor calculation mistakes set aside, shouldn't it be the same number of users?

Thank you and have a nice weekend!!

@Christine_Domgoergen_WMDE I am on it. Prioritized on my Phab dashboard.

@Christine_Domgoergen_WMDE

  1. Table 1.1.2. could you give us three more decimal places here? Thank you.

Done.

  1. The numbers click rate mean + median and close rate mean and median from table 1.1.2 are really interesting for us. Could you also calculate them for the devices and add them to table 1.2.2 as well as for Banner Actions var vs. ctrl and add them to table 1.3.2. as well as for nt vs. cs and add them to table 1.4.2?

Done.

  1. Page Views per banner. I have a general question here: We had two distinctions: one into target groups (cs vs nt) and one into banners (ctrl vs. var). Since both target groups have seen banner var (non-variable text) we should be able to compare page views var vs. cs vs. nt - right? So I expect the page views of var + cs + nt to sum up to the total page views. However, when I sum up the page views of LP 1 nt=2089 + cs=1354 I already get the total page view number. Did I miss something? Is it possible to compare the page views var (both target groups) against ctrl.cs and ctrl.nt? Content-wise we had those three banners and we would like to know which of our three banners performed best.

Ok, let's go step by step here...

3.A We had two distinctions: one into target groups (cs vs nt) and one into banners (ctrl vs. var).

What you call "distinctions" are actually your experimental factors, i.e. the sources of your experimental manipulation.

3.B Since both target groups have seen banner var (non-variable text)...

Confusion is added by using var for non-variable text, as you explain, and ctrl (typical abbr. for control) for - I can only assume from you have explained - variable text...

3.C ... we should be able to compare page views var vs. cs vs. nt - right?

Well, yes. But "compare" in what sense? Chart them? We can do that. Precisely, we can compare var/cs vs. var/nt pageviews.

3.D So I expect the page views of var + cs + nt to sum up to the total page views.

No no - see 3.E.

3.E However, when I sum up the page views of LP 1 nt=2089 + cs=1354 I already get the total page view number. Did I miss something?

Yes, that would be expected, thanks for checking it out. Namely, each pageview in this campaign is produced either by an nt participant of by a cs participant, so it is necessary that nt + cs pageviews equals to the total sum of pageviews.

  1. F. Is it possible to compare the page views var (both target groups) against ctrl.cs and ctrl.nt? Content-wise we had those three banners and we would like to know which of our three banners performed best.

Yes, you had var seen by both cs and nt, and then ctrl.cs seen by cs and ctrl.nt seen by nt, and then we have an experimental design with var as a baseline and ctrl in two variants that is expected to have some effect above or below var. Nice. The comparison is possible, let me know if you would like to have it performed.

  1. Edit activity: which group of users did you use for the analysis of the edit activity: those who saw the banner or those who clicked on the banner? How many unique users are those?

Those who clicked on the banner, as suggested by @Verena. There were 681 users who made any edits.

  1. Edit classes: to which timeframe do the edit classes refer to? Is it for example 2-4 edits in total (before campaign + during campaign + after campaign) or is it per day/wek/month in 3.4.1 - 3.4.2B?

3.4.1, 3.4.2A, and 3.4.2B all report edit classes for: before campaign + during campaign + after campaign. 3.4.3 provides before campaign - during campaign - after campaign segments. The timeframes are readable from the charts (they all use the same time span, take a look).

  1. Edit classes 3.4.3 I summed up the number of users in the edit classes before, during and after the campaign and got a different number of users for each of the time spans: before 535, during 489 and after 427. Possible minor calculation mistakes set aside, shouldn't it be the same number of users?

Why would you expect the same number of users edited in three different time frames: before, during, and after campaign?

@GoranSMilovanovic Thank you, I will have a look at the report now! For your questions:

3.F.

Yes, you had var seen by both cs and nt, and then ctrl.cs seen by cs and ctrl.nt seen by nt, and then we have an experimental design with var as a baseline and ctrl in two variants that is expected to have some effect above or below var. Nice. The comparison is possible, let me know if you would like to have it performed.

Yes please, that would be great.

Why would you expect the same number of users edited in three different time frames: before, during, and after campaign?

Because the edit classes include 0 edits in the first edit class (0-1 edits), so I would expect that even if users stopped editing during or after the campaign they still should be included in the table.

@GoranSMilovanovic Just to be sure:

  1. Table 1.1.2: is the click_rate_mean and the click_rate_median really 0,09000 and the close_rate_median 0,35000?

@GoranSMilovanovic Did you have time to look at this yet? We are having a meeting on this tomorrow and it would be great to have the numbers. Thank you!

@Christine_Domgoergen_WMDE

Here's an updated Campaign Report for you:

Notes.

Yes, you had var seen by both cs and nt, and then ctrl.cs seen by cs and ctrl.nt seen by nt, and then we have an experimental design with var as a baseline and ctrl in two variants that is expected to have some effect above or below var. Nice. The comparison is possible, let me know if you would like to have it performed. - Yes please, that would be great.

Done. Take a look at section 2.5 Pageviews: Var/Ctrl vs. cs/nt.

Why would you expect the same number of users edited in three different time frames: before, during, and after campaign? - Because the edit classes include 0 edits in the first edit class (0-1 edits), so I would expect that even if users stopped editing during or after the campaign they still should be included in the table.

Got it, fixed in 3.4.3 Edit Classes: before, during, and after the campaign.

Table 1.1.2: is the click_rate_mean and the click_rate_median really 0,09000 and the close_rate_median 0,35000?

Yes, they are. Precisely,

  • we first compute the click and close rate on each campaign day (these numbers are shown in Chart 1.1.2 Daily Banner Click and Close rates, aggregated across the campaign banners);
  • the reported means and medians are derived from the daily click and close rates.

@GoranSMilovanovic Great, thank you for the quick update! In our meeting yesterday two more questions came up:

3.4.3. Edit Classes: So in total 681 Users edited and now the nubers sum up nicely to exact that amount in all time periods (before, during and after the campaigns). Actually it would be interesting to see how many of those in the edit classes 0-1 did not edit before/during and after the campaign. Can you split the group up in 0 Edits and 1 Edit?

autochecked users? It would be interesting to see if the users who clicked on the banner are part of the user group "autochecked user" (https://de.wikipedia.org/w/index.php?title=Spezial:Benutzer&group=autoreview). Is it possible to analyze how many were part of the group before the campaign, during the campaign and after the campaign, split up in cohorts nt and cs? Can you find this data in the database? In dewp this is a marker for a user group with certain rights, like admins for example.

@Christine_Domgoergen_WMDE

3.4.3. Edit Classes: So in total 681 Users edited and now the nubers sum up nicely to exact that amount in all time periods (before, during and after the campaigns). Actually it would be interesting to see how many of those in the edit classes 0-1 did not edit before/during and after the campaign. Can you split the group up in 0 Edits and 1 Edit?

See section 3.4.4. in the following update:

autochecked users? It would be interesting to see if the users who clicked on the banner are part of the user group "autochecked user" (https://de.wikipedia.org/w/index.php?title=Spezial:Benutzer&group=autoreview). Is it possible to analyze how many were part of the group before the campaign, during the campaign and after the campaign, split up in cohorts nt and cs? Can you find this data in the database? In dewp this is a marker for a user group with certain rights, like admins for example.

@Tobi_WMDE_SW @Christine_Domgoergen_WMDE

This calls for a re-factoring of the back-end ETL data gathering procedures and then the development of the analytics code. Given that it will change the structure of the datasets that we use to analyze the banner campaigns, it needs to be developed as a stand-alone module and not entered as modification to the existing code for this campaign. In other words, it will take some time, because I have to focus on Wikidata related tasks in the following days.

@GoranSMilovanovic

See section 3.4.4. in the following update: 2020_WMDE_OccasionalEditorsBannerCampaign .nb.html3 MBDownload

Great, thank you!

This calls for a re-factoring of the back-end ETL data gathering procedures and then the development of the analytics code. Given that it will change the structure of the datasets that we use to analyze the banner campaigns, it needs to be developed as a stand-alone module and not entered as modification to the existing code for this campaign. In other words, it will take some time, because I have to focus on Wikidata related tasks in the following days.

Okay, this is not urgent, you can prioritize other tasks, no problem!

One other question about the click rate. I've been wondering about how the click rate of 9 percent is calculated. It comes from the schema, right? Because normally I would have had a look at the page views (3443) and the impressions (13.500) and then calculated page views/impressions and get to a number of 25,5 percent. Am I making a mistake or is there any explanation for the difference? Thank you!

@Christine_Domgoergen_WMDE

Okay, this is not urgent, you can prioritize other tasks, no problem!

I will prioritize it and as soon as I can. I really have an increasing focus on the Wikidata tasks these days.

One other question about the click rate. I've been wondering about how the click rate of 9 percent is calculated. It comes from the schema, right? Because normally I would have had a look at the page views (3443) and the impressions (13.500) and then calculated page views/impressions and get to a number of 25,5 percent. Am I making a mistake or is there any explanation for the difference? Thank you!

The mean click rate (click_mean_rate in Table 1.1.2) that you are talking about - the 9% as observed in Table 1.1.2 Mean and Median Click and Close rates (across days) in the Report, is calculated in the following way:

  • we take the data from Chart 1.1.2 Daily Banner Click and Close rates, aggregated across the campaign banners just above the table,
  • these data showing us the daily banner click and close rates, aggregated across the campaign banners,
  • and then we calculate the mean (and the median) click rate across the campaign days.

The data are from the event.WMDEBannerActions schema.

So, the click rate here is not calculated as pageviews/banner impressions as in previous campaigns - because we have the data from the event.WMDEBannerActions that we tested in the preparations for this campaign (see T250791#6128795).

When we take together all the daily data for all the campaign banners from the event.WMDEBannerActions schema and aggregate (i.e. sum all daily observations per campaign banner), these are the click rates that we get to observe:

   bannername                                seen_by clicked_by click_rate
 1 WMDE_oceditors_spring_2020_ctrl_cs           1352        123     0.0910
 2 WMDE_oceditors_spring_2020_ctrl_ipad_cs       366         54     0.148 
 3 WMDE_oceditors_spring_2020_ctrl_ipad_nt       709         70     0.0987
 4 WMDE_oceditors_spring_2020_ctrl_mobile_cs     431         98     0.227 
 5 WMDE_oceditors_spring_2020_ctrl_mobile_nt    1287        196     0.152 
 6 WMDE_oceditors_spring_2020_ctrl_nt           2774        172     0.0620
 7 WMDE_oceditors_spring_2020_var_cs            1330        138     0.104 
 8 WMDE_oceditors_spring_2020_var_ipad_cs        351         49     0.140 
 9 WMDE_oceditors_spring_2020_var_ipad_nt        688         84     0.122 
10 WMDE_oceditors_spring_2020_var_mobile_cs      455         60     0.132 
11 WMDE_oceditors_spring_2020_var_mobile_nt     1167        150     0.129 
12 WMDE_oceditors_spring_2020_var_nt            2590        148     0.0571

@GoranSMilovanovic Okay, thank you. I still don't understand why there is such a big difference. I understand the rate is calculated on the basis of the numbers in "clicked_by", but why are they in sum so different from the number of page views from the database? And when I divide the clicked_by/seen_by I get a click_rate of 9,94% and not 9 percent straight. Maybe @kai.nissen can help here?

GoranSMilovanovic added a comment.EditedJul 28 2020, 9:24 AM

@Christine_Domgoergen_WMDE I guess you get 9,94% from the table in T249617#6337564, because if you use the daily data to calculate the mean and median per banner (i.e. the data from Chart 1.1.2 Daily Banner Click and Close rates, aggregated across the campaign banners in the Report) then the number is 9% as reported.

However, there should not be a difference between (A) the banner click rates calculated as I have explained in T249617#6337564 from the event.WMDEBannerActions schema and (B) calculated in the same way from the pageviews/impressions ratio.

@Christine_Domgoergen_WMDE When you first tried to calculate the click rates from the pageviews, what banner impressions data did you use: those reported from the event.WMDEBannerActions schema (used in the Report) or those from the Banner Impressions tab in the [[ https://docs.google.com/spreadsheets/d/1p2-cp3KvbWN68Q9SvCpC5v6BncMyJcIzfdQv4jE49qQ/edit#gid=1078203925 | Campaign spreadsheet? I am asking because I remember @kai.nissen explaining that the two are not expected to be compatible.

I would suggest the following steps:

  1. First decide exactly how do we want the click rates calculated: calculating means and medians from daily per banner impressions, or calculating means and medians from the aggregated dataset;
  2. then calculate the click rates as described in (A) and (B) above;
  3. see how large the discrepancy is and why does it discrepancy occur.

Again, there should be no difference at all: if each click implies a pageview, as we assumed in all our previous campaigns... the numbers should be exactly the same.

@GoranSMilovanovic Thank you for the explanations.

So the 9,94% come from the data in the spreadsheet (tab banner interactions NOT banner impressions): sum of "seen_by"/sum of "clicked_by". Shouldn't this be the same to when I use the data from Chart 1.1.2? (still confused...) And iIf I calculate the mean from "click_rate" I even get 11,923%...

However, there should not be a difference between (A) the banner click rates calculated as I have explained in T249617#6337564 from the event.WMDEBannerActions schema and (B) calculated in the same way from the pageviews/impressions ratio.

No and I am confused how to solve this. Could it be that the schema counts clicked_by differently? Is there a difference made for unique users for example in either, the "clicked_by" or the page views from the number of page views you get from the data set?

First decide exactly how do we want the click rates calculated: calculating means and medians from daily per banner impressions, or calculating means and medians from the aggregated dataset;

So to be sure: "calculating means and medians from daily per banner impressions" --> this is the data from the schema? And "aggregated dataset" would be the data you get from the database directly, like the page views?

GoranSMilovanovic added a comment.EditedJul 28 2020, 2:16 PM

@Christine_Domgoergen_WMDE

So the 9,94% come from the data in the spreadsheet (tab banner interactions NOT banner impressions): sum of "seen_by"/sum of "clicked_by".

I guess you mean the opposite: sum of "clicked_by"/sum of "seen_by"., which turns out to be 0.09940740741 or 9.94% as you have reported.

Shouldn't this be the same to when I use the data from Chart 1.1.2? (still confused...)

They are exactly the same, calculated from the source data used to produce Chart 1.1.2. The reason you get 0.09940740741 and not 0.09 as reported in Table 1.1.2 is because the table numbers are calculated from data already aggregated per each day to produce Chart 1.1.2. The reason why it is done that way is that because the Campaign Doc specifies that we need:

How do click and closing rate for each of the banners develop during the campaign? (rates per day)

and then I've used the per day banner click and closing rates to compute means and medians. The number 9.94% that you report is not a mean, or a median, of anything: it is the de facto click rate observed across all banner, across all days.

So, in order to finally resolve the confusion on the numbers in the Report, I've added Table 1.1.2B** De Facto Click and Close rates in percents which now encompasses the de facto click and close rates and matches your numbers exactly:

No and I am confused how to solve this. Could it be that the schema counts clicked_by differently? Is there a difference made for unique users for example in either, the "clicked_by" or the page views from the number of page views you get from the data set?

I have no idea since I am not responsible for the content of our databases and I do not even know how do people who are put the data there. Once again, it sounds logical that the clicked_by/seen_by ratio as computed from the event.WMDEBannerActions schema and the pageviews/seen_by ratio should be the one and the same number - simply because we expect clicked_by to be equal to pageviews.

So to be sure: "calculating means and medians from daily per banner impressions" --> this is the data from the schema? And "aggregated dataset" would be the data you get from the database directly, like the page views?

This is already explained above.

@GoranSMilovanovic Since we still have no idea why the ratios from the schema and the page view/seen by numbers are different I want to get some feedback from @kai.nissen about this before we close the ticket. Kai will be back from vacation in two weeks and I hope we can close the ticket when he has responded.

@GoranSMilovanovic Hi Goran, so I talked to Kai and he also cannot really explain, why the number of seen by is not equal the number of page views from the database. However he says in doubt the numbers you got from the database is more likely to be correct. Could you adapt the final report accordingly? Thank you! And afterwards we can finally close the tickent :-)

GoranSMilovanovic added a comment.EditedSep 12 2020, 3:05 PM

@Christine_Domgoergen_WMDE @kai.nissen @WMDE-leszek

Here is the situation with the campaign data:

  • I do have all the banner impressions data from May 14 to May 28 (when the campaign was run), aggregated per (a) campaign banner, (b) device, and (c) date, but I also remember @kai.nissen saying that these banner impressions data (that we get directly from the /beacon/impression path in wmf.webrequest) are approximate only;
  • I can recalculate the banner click rates from these impressions (click rate = pageviews_from_banner/impressions_from_banner), which is what we did for all our campaigns and before the introduction of event.WMDEBannerActions schema which is now said to be provide data of uncertain quality; in this scenario we simply assume that each banner click certainly generates a pageview;
  • If we choose to do this, the Campaign Report Sections 1 and 2 will need to undergo a thorough rework;
  • Also, there is a remaining question of what do we want to do with the event.WMDEBannerActions schema data on banner interactions that we now have in the campaign report (the latest report update is found in this ticket: T249617#6341009): completely remove anything related to banner interactions?

To motivate this a bit more, let's get back to the problem that we are facing. In what I am about to explain:

  • pageviews is the number of pageviews generated from a particular campaign banner on any campaign page; the source of this number is the wmf.webrequest table;
  • clicked_by and seen_by represent number of users who have clicked and/or seen a particular campaign banner, respectively; the source of these two numbers is the new event.WMDEBannerActions schema.

Now:

  • pageviews/seen_by should calculate the banner click rate, as well as
  • clicked_by/seen_by;
  • these two numbers should be exactly the same, but in this campaign they are not.

What I am worried about is the following:

  • if the clicked_by number from the new event schema is wrong, then pageviews/seen_by is what we need to use to calculate banner click rates, but
  • if the seen_by number from the new event schema is wrong, then even pageviews/seen_by does not work obviously.

Hence the question: how much do we trust the new event.WMDEBannerActions schema? Do we want (A) to rely on it completely, or do we want to (B) rely on the assumption that its seen_by field reports the banner impressions data correctly while its clicked_by field reports incorrect data, or (C) do we want to rely on this schema at all?

@kai.nissen Kai, can we get your opinion on this question:

Hence the question: how much do we trust the new event.WMDEBannerActions schema? Do we want (A) to rely on it completely, or do we want to (B) rely on the assumption that its seen_by field reports the banner impressions data correctly while its clicked_by field reports incorrect data, or (C) do we want to rely on this schema at all?

As I understood it, it is better to trust the seen_by data from the schema because the banner impressions from the wmf.webrequest table does just count all banner impressions before the targeting filter is applied. Correct? This would mean option (B) would be what we go for. Or might there be an error in the "seen_by" data as well which would maybe explain the very very high conversion rate of about 25 percent, which we get when we calculate pageviews from wmf.webrequest / seen_by from the schema?

Yes, Central Notice tracks that a banner was delivered to the browser. The banner code may decide not to show the banner and triggers a banner-seen event only if the banner is really displayed.

Thank you Kai!

@GoranSMilovanovic This means we go with option B!

@Christine_Domgoergen_WMDE

(B) rely on the assumption that its seen_by field reports the banner impressions data correctly while its clicked_by field reports incorrect data.

This means that we will use the event.WMDEBannerActions schema and its seen_by field for for banner impressions, while the banner clicks will be inferred from pageviews relying on the assumption that each banner click necessarily produces a pageview.

Please confirm. Thank you.

@Christine_Domgoergen_WMDE then finally, from T249617#6433362:

Hi Goran, so I talked to Kai and he also cannot really explain, why the number of seen by is not equal the number of page views from the database. However he says in doubt the numbers you got from the database is more likely to be correct. Could you adapt the final report accordingly? Thank you! And afterwards we can finally close the tickent :-)

should I just remove everything on banner clicks from event.WMDEBannerActions schema from this report?

@GoranSMilovanovic Yes, but it would need to be replaced by the pageviews you counted from the wmf.webrequest table. And then all the ratios, where the banner clicks from the schema were used need to be recalculated also: all the click rates etc. Could you do that?

@Christine_Domgoergen_WMDE I think so, yes. I will make sure that it is done before the onset of the October 2020 campaign.

@GoranSMilovanovic Hi Goran, do you have an update on this? Thank you :-)

@Christine_Domgoergen_WMDE Oh I almost forgot about this one and I knew there was something unanswered:

  • given that the decision is to use the event.wmdebannerinteractions table for banner clicks and closing the banners in the campaign that we are currently running (T262534), do we still want to change the existing report for this campaign (based on the same schema) or not? Are we about to run into inconsistencies in the long run if we start doing such things?

@GoranSMilovanovic Yes please. In the other campaign we have an offwiki landingpage so the data from the schema is our only source. In this campaign we have better data from the wmf.webrequest table, thats why we stick with this.

@Christine_Domgoergen_WMDE @WMDE-leszek Here is the re-work of this campaign's final report:

  • pageviews data replace clicked_by from event.wmdebannerinteractions everywhere;
  • it is indicated by a NOTE in this report whenever it was not possible to fully reproduce an analysis because it relied on per user observations from event.wmdebannerinteractions which we do not have in the pageviews dataset.

@GoranSMilovanovic Hi Goran, thank youu! I just checked the report and came across some differences in the impression numbers. In the second last version of the report the numbers of impressions and closed-by in the tables 1.1.A and 1.2.1 A are different to the numbers in this last version (13500 vs. 67500). Why is that? Could you check this? Thank you!

GoranSMilovanovic added a comment.EditedOct 15 2020, 12:31 PM

@Christine_Domgoergen_WMDE

In the second last version of the report the numbers of impressions and closed-by in the tables 1.1.A and 1.2.1 A are different to the numbers in this last version (13500 vs. 67500).

I will check it out, that should not have changed.

@Christine_Domgoergen_WMDE A missing aggregate from the re-work of this report has caused this problem:

In the second last version of the report the numbers of impressions and closed-by in the tables 1.1.A and 1.2.1 A are different to the numbers in this last version (13500 vs. 67500).

Please find the corrected report attached. The numbers under discussion now match what was initially reported. Thanks for catching!

@Christine_Domgoergen_WMDE Do we need anything else in relation to this ticket? Resolved?

Also, @Aklapper has noticed - I remember seeing an email yesterday or so - that there are some very, very old WMDE campaign related tickets still open.

@GoranSMilovanovic I just had a quick glance at the report and it looks good, so we can resolve the ticket yes! I checked on the other tickets, thank you.