Page MenuHomePhabricator

Create test banner for tracking test
Closed, ResolvedPublic

Description

After the scheme ist deployed a test banner has to be created to check functionality of the scheme

related tasks: https://phabricator.wikimedia.org/T250791

Event Timeline

@Christine_Domgoergen_WMDE, @Verena Current banner status is deployed to Central Notice for testing:
WMDE_oceditors_spring_2020_a
WMDE_oceditors_spring_2020_b

There is an issue with the close button, but I think you can already start testing.

Thank you Kai. It seems that the campaign tags have changed. Could you please provide the new campaign tags for all banners?

@GoranSMilovanovic
We just tested by clicking on the banners with two different users who fulfill the targeting requirements:
https://de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement?campaign=WMDE_oceditors_spring_2020_var_cs
https://de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement?campaign=WMDE_oceditors_spring_2020_ctrl_cs

Could you check if you can find user ids for the event?

Sure:

WMDE_oceditors_spring_2020_ctrl_nt
WMDE_oceditors_spring_2020_ctrl_ipad_nt
WMDE_oceditors_spring_2020_ctrl_mobile_nt
WMDE_oceditors_spring_2020_ctrl_nt
WMDE_oceditors_spring_2020_ctrl_ipad_nt
WMDE_oceditors_spring_2020_ctrl_mobile_nt
WMDE_oceditors_spring_2020_var_cs
WMDE_oceditors_spring_2020_var_ipad_cs
WMDE_oceditors_spring_2020_var_mobile_cs
WMDE_oceditors_spring_2020_var_cs
WMDE_oceditors_spring_2020_var_ipad_cs
WMDE_oceditors_spring_2020_var_mobile_cs

Suffix explanation:

  • ctrl is the banner that dynamically displays text depending on the target group
  • var is the banner that shows the same text for both target groups
  • ipad/mobile does not represent the actual device type, but the display mode of the banner (small desktop screens may be reported as ipad)
  • cs/nt indicates, which target group the user belongs to

@Verena

Could you check if you can find user ids for the event?

select dt, event.bannerName, event.bannerAction, event.bannerImpressions, event.userID from event.wmdebannerinteractions where year=2020 and month=5 and (day=11 or day=12 or day=13);

(i.e. all that we have in event.wmdebannerinteractions from May 11 until now) results in:

dt                      bannername                              banneraction    bannerimpressions       
2020-05-11T16:18:09Z    WMDE_oceditors_spring_2020_mobile       banner-closed   30      
2020-05-11T16:18:09Z    WMDE_oceditors_spring_2020_mobile       banner-seen     30      
2020-05-12T17:01:07Z    WMDE_oceditors_spring_2020_mobile       banner-seen     2       
2020-05-12T17:01:16Z    WMDE_oceditors_spring_2020_mobile       banner-seen     3       
2020-05-12T17:01:28Z    WMDE_oceditors_spring_2020_mobile       banner-seen     4       
2020-05-12T17:01:03Z    WMDE_oceditors_spring_2020_mobile       banner-seen     1       
2020-05-12T17:02:20Z    WMDE_oceditors_spring_2020_mobile       banner-seen     5       
2020-05-12T08:52:12Z    WMDE_oceditors_spring_2020_mobile       banner-seen     2       
2020-05-12T08:51:26Z    WMDE_oceditors_spring_2020_mobile       banner-seen     1       
2020-05-12T08:51:24Z    WMDE_oceditors_spring_2020_mobile       banner-seen     31      
2020-05-12T08:51:26Z    WMDE_oceditors_spring_2020_mobile       banner-clicked  1       
11 rows selected (0.363 seconds)

Note. UserIDs removed from the output.

@GoranSMilovanovic
Yes, Hive tables get updated with quite a delay. Do you know, if we can look up event data somewhere before it is fed to Hive?

I just noticed that there are log entries from today's tests:

dt			bannername				banneraction	bannerimpressions
2020-05-13T07:40:25Z	WMDE_oceditors_spring_2020_mobile	banner-seen	32
2020-05-13T08:55:15Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-seen	2
2020-05-13T08:55:00Z	WMDE_oceditors_spring_2020_var_nt	banner-seen	2
2020-05-13T08:54:24Z	WMDE_oceditors_spring_2020_var_nt	banner-seen	1
2020-05-13T08:55:00Z	WMDE_oceditors_spring_2020_var_nt	banner-clicked	2
2020-05-13T08:54:28Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-seen	1
2020-05-13T08:54:50Z	WMDE_oceditors_spring_2020_var_nt	banner-clicked	1
2020-05-13T08:54:45Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-clicked	1
2020-05-13T08:55:47Z	WMDE_oceditors_spring_2020_var_nt	banner-seen	3
2020-05-13T09:10:05Z	WMDE_oceditors_spring_2020_var_cs	banner-clicked	1
2020-05-13T09:23:39Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-seen	4
2020-05-13T09:38:39Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-clicked	6
2020-05-13T09:40:31Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-seen	8
2020-05-13T09:37:42Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-seen	5
2020-05-13T09:38:39Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-seen	6
2020-05-13T09:15:54Z	WMDE_oceditors_spring_2020_ctrl_cs	banner-clicked	2
2020-05-13T09:41:16Z	WMDE_oceditors_spring_2020_var_mobile_nt	banner-seen	1
2020-05-13T09:16:14Z	WMDE_oceditors_spring_2020_var_cs	banner-clicked	1
2020-05-13T09:40:37Z	WMDE_oceditors_spring_2020_ctrl_mobile_nt	banner-seen	1
2020-05-13T09:10:32Z	WMDE_oceditors_spring_2020_ctrl_cs	banner-clicked	1
2020-05-13T09:40:56Z	WMDE_oceditors_spring_2020_ctrl_mobile_nt	banner-seen	2
2020-05-13T09:40:31Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-clicked	8
2020-05-13T09:23:39Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-closed	4
2020-05-13T09:40:11Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-seen	7
2020-05-13T09:02:00Z	WMDE_oceditors_spring_2020_ctrl_cs	banner-seen	1
2020-05-13T09:37:51Z	WMDE_oceditors_spring_2020_ctrl_ipad_nt	banner-seen	1
2020-05-13T09:09:25Z	WMDE_oceditors_spring_2020_var_cs	banner-seen	1
2020-05-13T09:16:13Z	WMDE_oceditors_spring_2020_var_cs	banner-seen	1
2020-05-13T09:08:55Z	WMDE_oceditors_spring_2020_ctrl_cs	banner-seen	1
2020-05-13T09:15:54Z	WMDE_oceditors_spring_2020_ctrl_cs	banner-seen	2
2020-05-13T09:23:26Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-clicked	3
2020-05-13T09:23:26Z	WMDE_oceditors_spring_2020_ctrl_nt	banner-seen	3

@kai.nissen Please, what you have reported in T251535#6133536 is from the event.WMDEBannerInteractions Hive table - or have you used a different source? If yes, well, then it really takes some time until everything is in hdfs.

@Christine_Domgoergen_WMDE I will test the pageviews in T251535#6133655 as soon as I figure out what is happening with both the pageviews and user registrations in T249617.

@kai.nissen Confirming your T251535#6133536:

select dt, event.bannerName, event.bannerAction, event.bannerImpressions, event.userID from event.wmdebannerinteractions where year=2020 and month=5 and (day=11 or day=12 or day=13);

results in:

dt                      bannername                              banneraction    bannerimpressions       
2020-05-11T16:18:09Z    WMDE_oceditors_spring_2020_mobile       banner-closed   30      
2020-05-11T16:18:09Z    WMDE_oceditors_spring_2020_mobile       banner-seen     30      
2020-05-12T17:01:07Z    WMDE_oceditors_spring_2020_mobile       banner-seen     2       
2020-05-12T17:01:16Z    WMDE_oceditors_spring_2020_mobile       banner-seen     3       
2020-05-12T17:01:28Z    WMDE_oceditors_spring_2020_mobile       banner-seen     4       
2020-05-12T17:01:03Z    WMDE_oceditors_spring_2020_mobile       banner-seen     1       
2020-05-12T17:02:20Z    WMDE_oceditors_spring_2020_mobile       banner-seen     5       
2020-05-12T08:52:12Z    WMDE_oceditors_spring_2020_mobile       banner-seen     2       
2020-05-12T08:51:26Z    WMDE_oceditors_spring_2020_mobile       banner-seen     1       
2020-05-12T08:51:24Z    WMDE_oceditors_spring_2020_mobile       banner-seen     31     
2020-05-12T08:51:26Z    WMDE_oceditors_spring_2020_mobile       banner-clicked  1       
2020-05-13T07:40:25Z    WMDE_oceditors_spring_2020_mobile       banner-seen     32     
2020-05-13T08:55:15Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-seen     2       
2020-05-13T08:55:00Z    WMDE_oceditors_spring_2020_var_nt       banner-seen     2      
2020-05-13T08:54:24Z    WMDE_oceditors_spring_2020_var_nt       banner-seen     1      
2020-05-13T08:55:00Z    WMDE_oceditors_spring_2020_var_nt       banner-clicked  2     
2020-05-13T08:54:28Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-seen     1       
2020-05-13T08:54:50Z    WMDE_oceditors_spring_2020_var_nt       banner-clicked  1      
2020-05-13T08:54:45Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-clicked  1       
2020-05-13T08:55:47Z    WMDE_oceditors_spring_2020_var_nt       banner-seen     3       
2020-05-13T09:10:05Z    WMDE_oceditors_spring_2020_var_cs       banner-clicked  1      
2020-05-13T09:23:39Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-seen     4       
2020-05-13T09:38:39Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-clicked  6      
2020-05-13T09:40:31Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-seen     8       
2020-05-13T09:37:42Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-seen     5       
2020-05-13T09:38:39Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-seen     6       
2020-05-13T09:15:54Z    WMDE_oceditors_spring_2020_ctrl_cs      banner-clicked  2      
2020-05-13T09:41:16Z    WMDE_oceditors_spring_2020_var_mobile_nt        banner-seen     1       
2020-05-13T09:16:14Z    WMDE_oceditors_spring_2020_var_cs       banner-clicked  1       
2020-05-13T09:40:37Z    WMDE_oceditors_spring_2020_ctrl_mobile_nt       banner-seen     1       
2020-05-13T09:10:32Z    WMDE_oceditors_spring_2020_ctrl_cs      banner-clicked  1       
2020-05-13T09:40:56Z    WMDE_oceditors_spring_2020_ctrl_mobile_nt       banner-seen     2       
2020-05-13T09:40:31Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-clicked  8       
2020-05-13T09:23:39Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-closed   4       
2020-05-13T09:40:11Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-seen     7       
2020-05-13T09:02:00Z    WMDE_oceditors_spring_2020_ctrl_cs      banner-seen     1       
2020-05-13T09:37:51Z    WMDE_oceditors_spring_2020_ctrl_ipad_nt banner-seen     1   
2020-05-13T09:09:25Z    WMDE_oceditors_spring_2020_var_cs       banner-seen     1      
2020-05-13T09:16:13Z    WMDE_oceditors_spring_2020_var_cs       banner-seen     1      
2020-05-13T09:08:55Z    WMDE_oceditors_spring_2020_ctrl_cs      banner-seen     1       
2020-05-13T09:15:54Z    WMDE_oceditors_spring_2020_ctrl_cs      banner-seen     2       
2020-05-13T09:23:26Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-clicked  3      
2020-05-13T09:23:26Z    WMDE_oceditors_spring_2020_ctrl_nt      banner-seen     3

@Christine_Domgoergen_WMDE As of the pages mentioned in T251535#6133655:

  • I can see the pageviews in wmf.webrequest,
  • but I can't see any campaign tags in the respective uri_query fields; I was specifically looking for anything similar to WMDE_oceditors_spring_2020.

This holds for 2020/05/12 and 2020/05/13 until now.

@kai.nissen Please, what you have reported in T251535#6133536 is from the event.WMDEBannerInteractions Hive table - or have you used a different source? If yes, well, then it really takes some time until everything is in hdfs.

Yes, it's from that Hive table.

@GoranSMilovanovic

but I can't see any campaign tags in the respective uri_query fields; I was specifically looking for anything similar to WMDE_oceditors_spring_2020.

So why could this be? It cannot be the patch because it is be in place in dewiki. We would need the tag in order to see from which banner which page views were produced, correct?

  • but I can't see any campaign tags in the respective uri_query fields; I was specifically looking for anything similar to WMDE_oceditors_spring_2020.

That's strange. Banner clicks result in requesting e. g. https://de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement?campaign=WMDE_oceditors_spring_2020_ctrl_nt.

@Christine_Domgoergen_WMDE When were the pages mentioned in T251535#6133655 tested (e.g. "seen") from a banner click?
@kai.nissen As of your comment in T251535#6134106, also: when did you test it?

I am asking because it is possible that the wmf.webrequest still does not have the data on your pageviews.

I tested just now to reproduce the behaviour. But I tested this morning and yesterday as well. Each banner-clicked event should produce an entry in wmf.webrequest including something like campaign=WMDE_oceditors_spring_2020%.

@GoranSMilovanovic just shortly before I made the comment, arount 2.30 pm

@kai.nissen @Christine_Domgoergen_WMDE Well the data should have entered the wmf.webrequest table by now, I guess.

Ok. Either it's my code (high probability) or there's something nasty happening in the background (low probability). Let me see.

@kai.nissen @Christine_Domgoergen_WMDE It's my R code orchestrating the HiveQL calls that is missing something, direct test shows the pageviews are present:

select dt, uri_host, uri_path, uri_query from wmf.webrequest where (uri_host = "de.wikipedia.org" or uri_host = "de.m.wikipedia.org") and (uri_path = "/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement" or uri_path = "/wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement/Literatur" or uri_path = "/wiki/Wikipedia:Mentorenprogramm") and (uri_query like "%WMDE_oceditors_spring_2020_%") and year = 2020 and month = 5 and day = 13;

results in:

dt      uri_host        uri_path        uri_query
2020-05-13T09:00:45Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T14:37:50Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T09:37:52Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_ipad_nt
2020-05-13T09:16:13Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_var_cs
2020-05-13T09:07:18Z    de.m.wikipedia.org      /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_3all
2020-05-13T09:40:31Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_nt
2020-05-13T09:19:20Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T08:59:48Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_3all
2020-05-13T13:49:26Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_nt
2020-05-13T14:39:11Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T14:29:22Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T07:36:58Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_ipad_cs
2020-05-13T08:54:50Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_var_nt
2020-05-13T08:55:00Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_var_nt
2020-05-13T08:54:45Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_nt
2020-05-13T09:51:58Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_3all
2020-05-13T09:07:21Z    de.m.wikipedia.org      /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_3all
2020-05-13T09:01:14Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_3all
2020-05-13T09:23:26Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_nt
2020-05-13T09:07:03Z    de.m.wikipedia.org      /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_3all
2020-05-13T09:15:54Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T09:13:07Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_var_cs
2020-05-13T09:40:32Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_var_cs
2020-05-13T09:38:39Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_nt
2020-05-13T09:10:32Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T09:13:10Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T09:16:12Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_3all
2020-05-13T09:10:05Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_var_cs
2020-05-13T09:42:21Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_ctrl_cs
2020-05-13T09:18:57Z    de.wikipedia.org        /wiki/Wikipedia:Wikimedia_Deutschland/DeinEngagement    ?campaign=WMDE_oceditors_spring_2020_var_cs
30 rows selected (233.081 seconds)

So we're safe except that for I need to improve my R skills, which will take place in the following hours. Reporting back as soon as the analytics procedure is corrected and in place. Thank you.

@Christine_Domgoergen_WMDE @Verena @kai.nissen

  • Analytics code is fixed;
  • putting everything together now: (a) banner impressions, (b) banner actions, (c) pageviews, (d) user edits;
  • deploying the code on stat1004 on a daily update schedule;
  • campaign daily reporting is on as of tomorrow.

Thank you for testing!