Page MenuHomePhabricator

Use EventLogging to log banner interactions
Open, Needs TriagePublic2 Estimated Story Points

Description

We want to know how users interact with the banners and how many impressions users have seen before interacting with the banner. We can use Schema:WMDEBannerInteractions to do that, but need to extend it to also be able to log banner impressions.

The schema has been used previously, but never logged data in the respective Hive table. We need to find the reason and fix that.

Acceptance Criteria

  • Banner trigger EventLogging for the following cases:
    • Banner impressions using the newly added banner action banner-seen
    • Banner clicks
    • Banner close button clicks
  • The user ID is added to the event schema.
  • The banner name takes the device type into account (mobile, desktop, ipad)
  • The banner name also contains a suffix that can be used to distinguish between banner 1 and 2
  • The changed schema revision is deployed during a SWAT deployment.

Notes

Event Timeline

Restricted Application added a project: WMDE-FUN-Team. · View Herald TranscriptApr 21 2020, 11:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
kai.nissen updated the task description. (Show Details)Apr 21 2020, 11:49 AM
kai.nissen updated the task description. (Show Details)Apr 21 2020, 11:59 AM
kai.nissen set the point value for this task to 13.
kai.nissen added a comment.EditedApr 27 2020, 10:02 AM

I tested it using a copy of a past campaign. Error log entry is on Kibana

kai.nissen updated the task description. (Show Details)Apr 27 2020, 10:07 AM
kai.nissen updated the task description. (Show Details)Apr 27 2020, 10:13 AM
kai.nissen changed the point value for this task from 13 to 8.
kai.nissen updated the task description. (Show Details)Apr 28 2020, 1:27 PM

Change 593502 had a related patch set uploaded (by Abban Dunne; owner: Abban Dunne):
[mediawiki/extensions/WikimediaEvents@master] Update Banner Interaction Schema

https://gerrit.wikimedia.org/r/593502

Change 594693 had a related patch set uploaded (by Gabriel Birke; owner: Abban Dunne):
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.30] Update Banner Interaction Schema

https://gerrit.wikimedia.org/r/594693

Change 594694 had a related patch set uploaded (by Gabriel Birke; owner: Abban Dunne):
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.31] Update Banner Interaction Schema

https://gerrit.wikimedia.org/r/594694

Change 593502 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Update Banner Interaction Schema

https://gerrit.wikimedia.org/r/593502

Change 594694 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.31] Update Banner Interaction Schema

https://gerrit.wikimedia.org/r/594694

Change 594693 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.30] Update Banner Interaction Schema

https://gerrit.wikimedia.org/r/594693

Mentioned in SAL (#wikimedia-operations) [2020-05-11T11:23:25Z] <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.35.0-wmf.31/extensions/WikimediaEvents/: SWAT: [[gerrit:594694|Update Banner Interaction Schema (T250791, wmf.31)]] (duration: 01m 07s)

Mentioned in SAL (#wikimedia-operations) [2020-05-11T11:30:12Z] <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.35.0-wmf.30/extensions/WikimediaEvents/: SWAT: [[gerrit:594693|Update Banner Interaction Schema (T250791, wmf.30)]] (duration: 01m 08s)

kai.nissen changed the point value for this task from 8 to 2.May 11 2020, 1:10 PM

I tested this with a slightly modified version of the test banner. In order to test for errors, please have a look at my example query.

@kai.nissen @Verena @WMDE-leszek I have tested the revised WMDEBannerInteractions schema:

select dt, event.bannerName, event.bannerAction, event.bannerImpressions, event.userID from event.wmdebannerinteractions where year=2020 and month=05 and (day=10 or day=11) limit 100;

results in (userid field values stripped, while both records refer to the one and same user):

dt                      bannername                              banneraction    bannerimpressions       userid
2020-05-11T16:18:09Z    WMDE_oceditors_spring_2020_mobile       banner-closed   30      
2020-05-11T16:18:09Z    WMDE_oceditors_spring_2020_mobile       banner-seen     30

Q1. First row: does this mean that a particular user has seen the banner 30 times before closing the banner on 2020-05-11T16:18:09Z?
Q2. Second: does this mean that a particular user has received the 30th banner impression on 2020-05-11T16:18:09Z?

Also,

Q3. The campaign Central Notice setup defines different weights (see section: Assigned banners) for banner A, B, and C: I guess these numbers refer to banner impression rates?

Q4. From the ticket description: "The banner name also contains a suffix that can be used to distinguish between banner 1 and 2" - I do not see a suffix in the banner names in these test results, and also the campaign doc and Central Notice setup mention three banners (A, B, C), while only two (A, B) are mentioned in the ticket description here.

And finally,

Q5. In T249617 we find:

The a/b test scope is comparing target group specific banners (1 and 2) against banner 3:
banner 1: target group A,
banner 2: target group B,
banner 3: target group A+B

So, banner 1 will be presented to group A, banner 2 will be presented to group B, but banner 3 will be seen by both groups: is it ensured that an editor from group A who sees banner 1 does not also see banner 3, and that an editor in group B who sees banner 2 does not also see banner 3?

Thank you.

Q1. First row: does this mean that a particular user has seen the banner 30 times before closing the banner on 2020-05-11T16:18:09Z?
Q2. Second: does this mean that a particular user has received the 30th banner impression on 2020-05-11T16:18:09Z?

Both is correct.

Q3. The campaign Central Notice setup defines different weights (see section: Assigned banners) for banner A, B, and C: I guess these numbers refer to banner impression rates?

The setup will be changed before starting. It will consist of two equally weighted banners.

Q4. From the ticket description: "The banner name also contains a suffix that can be used to distinguish between banner 1 and 2" - I do not see a suffix in the banner names in these test results, and also the campaign doc and Central Notice setup mention three banners (A, B, C), while only two (A, B) are mentioned in the ticket description here.

I just tested the event logging functionality yesterday. The test banner did not include suffixing the banner-name.

So, banner 1 will be presented to group A, banner 2 will be presented to group B, but banner 3 will be seen by both groups: is it ensured that an editor from group A who sees banner 1 does not also see banner 3, and that an editor in group B who sees banner 2 does not also see banner 3?

Yes. banner C will be in bucket B. Unless users clear their cookies, they will see only banner A/B or banner C.

@kai.nissen We're superfine to track and analyze the campaign then. Thank you very much.

Thank you all. @GoranSMilovanovic That means we are set and ready? Or is further testing before the campaign set recommended?

GoranSMilovanovic added a comment.EditedMay 12 2020, 5:47 PM

@Verena

Or is further testing before the campaign set recommended?

I would create a test account or two, just in case.

@Verena @Christine_Domgoergen_WMDE Given that the user registration procedure for our concurrently running campaign (see: T251524#6133796) fails to send the campaign tag to the event.serversideaccountcreation table, I would now strongly suggest to test user registrations for this campaign too.

@GoranSMilovanovic Since we are targeting only logged in users this is not necessary, we are lucky here :-)

GoranSMilovanovic added a subscriber: Addshore.EditedMay 13 2020, 3:46 PM

@Verena @Christine_Domgoergen_WMDE In relation to this campaign, please see: T251524#6133866 from @Addshore. Keyword: patch.

@Christine_Domgoergen_WMDE Oh, right... Thanks. I am beginning to get lost in concurrently testing two campaigns. @Verena please disregard my comment in T250791#6133879 (just above).