@MartinRulsch

Do we need separate subtasks?

As far as I can see, no, we do not, but please consult with @kai.nissen too.

Is the schedule okay with you?

The schedule is perfect with me, thank you.

@MartinRulsch

In order to test the campaign tracking for analytics, please let me know:

• what test banner is used,
• and what action has been taken (e.g. a pageview, a banner click, a user campaign registration, a revision made on behalf of a campaign registered user)

I will report back here as soon as any campaign related data are observed. Thanks.

@GoranSMilovanovic

1. test banner: https://meta.wikimedia.org/wiki/Special:CentralNoticeBanners/edit/WMDE_vssurvey_fall_2021_full
2. actions are twofold:

A) banner display, banner click, banner close related to the category where the banner is shown — although due to the script adjustments, maybe only the banner display ”banner-seen“ will be reported, see snippet:

var eventPayload = {
  $schema: '/analytics/legacy/wmdebannerinteractions/1.0.0',
  bannerName: thisCampaign, // dynamischer Wert abhängig von der Kategorie
  bannerAction: 'banner-seen', // banner-closed, banner-seen oder banner-clicked
};
var streamName = 'eventlogging_WMDEBannerInteractions';
mw.eventLog.submit( streamName, eventPayload ); //EventLoggin-Snippet

B) page-view of https://de.wikipedia.org/wiki/Wikipedia:Förderung with campaign tags

1. WMDE_vssurvey_fall_2021_lgbt
2. WMDE_vssurvey_fall_2021_chemie
3. WMDE_vssurvey_fall_2021_frau

as given in the external survey.

Is this sufficient? What else do you need? :)

One question is left on my part: When should I start the test banner (i.e., right now or when you give it a GO)?

@MartinRulsch

This is fine, except for that I would suggest testing banner interactions for all three campaign banners.

One question is left on my part: When should I start the test banner (i.e., right now or when you give it a GO)?

Whenever you like, the sooner the better because the camapaign onset is near, just let me know when you do it and then I will check out our databases to look for your test user behavior footprint there.

We only have one dynamic banner for this campaign which displays different content (text, link) according to the detected categories of the article.
The same detection also delivers the banner name for EventLogging so that we should be good with tracking each type of displayed banner:

bannerName: thisCampaign

Does that work for you?

As recommended, I've just enabled the campaign.

@MartinRulsch

I have no knowledge of how exactly the campaign setup works, but I will search for everything (banner interactions + pageviews, in the context of this campaign) related to the campaign banners.

Just please click the test banners, try to display as many different as possible... and that would do for the test phase. I guess.

@GoranSMilovanovic

The display of the banner, the links to all the surveys on wikimedia.de , the links from there to https://de.wikipedia.org/wiki/Wikipedia:F%C3%B6rderung (including campaign tags) all seem to be correct with me. Did you receive some data?

@MartinRulsch

Did you receive some data?

Nope, I will wait some time until I test later in the evening. The data are not transffered to our databases instantly. I will let you know here as soon as I have something.

Alright, thanks! :)

@MartinRulsch You are welcome : ) Stay in touch.

@MartinRulsch @kai.nissen

Test 1: event.wmdebannerinteractions data.
Test dates: 2021-09-29, 2021-09-30
Nothing was found on any of the following tags:

• WMDE_vssurvey_fall_2021_lgbt
• WMDE_vssurvey_fall_2021_chemie
• WMDE_vssurvey_fall_2021_frau

@GoranSMilovanovic Thanks for the info!

Afaics, that's okay as we have only ~12 survey entries in the test phase so far. Only few of them will click the link with the campaign tag.

What about the banner displays? Did you get any data of that from the WMDE schema?

@MartinRulsch

What about the banner displays? Did you get any data of that from the WMDE schema?

As explained in T291562#7390599: nothing from event.wmdebannerinteractions was found on the specified banners. That schema includes the seen value that indicates if the banner was delivered to a user or not. Thus there is no need to check any other schema (wmf.webrequest, for example).

Testing now for the page views of

https://de.wikipedia.org/wiki/Wikipedia:F%C3%B6rderung/Wikimedia_Deutschland/F%C3%B6rderung_bewerben

Alright, thanks for the explanation! That's a little bit unexpected because the banner is displayed to me for the right articles.

https://de.wikipedia.org/wiki/Wikipedia:F%C3%B6rderung/Wikimedia_Deutschland/F%C3%B6rderung_bewerben is just a placeholder link in the banner source code and is replaced with the right lime.wikimedia.de link according to the given article categories.

@MartinRulsch

As of T291562#7390618, you really need to get in touch with someone who understands the campaign setup as @kai.nissen
I can only tell you if something is present in the designated databases or not.

In the meantime,

Test 2. Pageviews
Test page: /wiki/Wikipedia:F%C3%B6rderung/Wikimedia_Deutschland/F%C3%B6rderung_bewerben
Test dates: 2021-09-29, 2021-09-30
On 2021-09-29, there were 12 pageviews of the test page, but none of them tagged with anything that contains WMDE_vssurvey_fall_2021.
There are still no pageviews registered for the test page on 2021-09-30.

@MartinRulsch
If required fields are missing from an event, the log entries don't make it to the tables. I just tested the banner and saw that the events were triggered. It takes ~2 hours for the triggered events to show up in the respective table, so I can update you on that later.

Two things I noticed when looking at the banner code:

• The banner is displayed before the banner code determines some of the content. Displaying the banner should happen after that.
• The banner-seen event is triggered in all cases, also if the article is not in one of the categories. Triggering the event should be depending on whether the banner was actually displayed or not. You should move the function call to mw.eventLog.submit() into the condition block that also displays the banner.

Hi @kai.nissen , thanks for the hints!

There's only definitions and the check of the right category before the condition:

if( ((LGBTisFound == true) || (ChemieisFound == true) || (FrauisFound == true)) && (mw.config.get('wgUserEditCount') > 200)) {

Everything below that is part of the condition, including the EventLoggin-Snippet and banner-seen. For me, this seems to meet your requirements in the second part. Or do you I mix things up here?

I've moved $('#wmde_foerderung_bewerben').css('display', 'block'); //die lesende Person befindet sich in einer der Zielkategorien, damit soll das Banner angezeigt werden to the end of the condition. I think that this meets your requirements for the first note, right?

Ugh, nevermind, I was apparently mixing things up... I thought, I had seen the event firing on the article about Helmut Kohl, which I assume is in neither of these categories.

Yes, moving the banner 'display', 'block' to the end of the condition was what I meant.

One more thing: The instrumentation docs seem to be wrong. In the event error logs I see a lot of error messages that suggest a different format for the event payload. I tested this in a cloned banner and it seems that this solves the issue.

So instead of:

var eventPayload = {
  $schema: '/analytics/legacy/wmdebannerinteractions/1.0.0',
  bannerName: thisCampaign, // dynamischer Wert abhängig von der Kategorie
  bannerAction: 'banner-seen', // banner-closed, banner-seen oder banner-clicked
  bannerImpressions: 0, // anzahl der banner impressions bei dieser aktion, muss in einem cookie gezählt werden, wenn überhaupt gewünscht
  userID: 0, // muss nicht verwendet werden
};

the event data needs to be encapsulated in an object called event, like this:

var eventPayload = {
  $schema: '/analytics/legacy/wmdebannerinteractions/1.0.0',
  event: {
    bannerName: thisCampaign, // dynamischer Wert abhängig von der Kategorie
    bannerAction: 'banner-seen', // banner-closed, banner-seen oder banner-clicked
    bannerImpressions: 0, // anzahl der banner impressions bei dieser aktion, muss in einem cookie gezählt werden, wenn überhaupt gewünscht
    userID: 0, // muss nicht verwendet werden
  }
};

Thanks, I've fixed it accordingly.

@MartinRulsch @kai.nissen

Thanks, I've fixed it accordingly.

Should I deploy the test, then?

@GoranSMilovanovic @kai.nissen

Good from my end. :)

@MartinRulsch

Just a reminder: did you interact with the campaign in any way since the most recent changes took place (e.g. a banner click, a page view)?

If yes, then: it is approx. 16:06 CET now; I will wait two hours and then run the test. Reporting back here, of course.

@GoranSMilovanovic Yes, there have been interactions with the banner since the most recent changes took place. Another test would be nice! Thanks in advance.

@MartinRulsch

The test date is 2021/09/30.

Banner Interactions test:

                        bannername seen_by closed_by mean_close_imp clicked_by
1 1 WMDE_vssurvey_fall_2021_chemie       3        NA             NA         NA
2 2   WMDE_vssurvey_fall_2021_frau      35        NA             NA         NA
3 3   WMDE_vssurvey_fall_2021_lgbt       1        NA             NA         NA
  mean_click_imp close_rate click_rate        day              campaign
1             NA         NA         NA 2021-09-30 VolunteerSupport_2021
2             NA         NA         NA 2021-09-30 VolunteerSupport_2021
3             NA         NA         NA 2021-09-30 VolunteerSupport_2021

As you can see, the campaign banners were registered in the eventLogging schema as seen, but no other interactions were registered (i.e. no closed_by, clicked_by actions were observed).

Pageviews test:

Seven (7) pageviews of https://de.wikipedia.org/wiki/Wikipedia:F%C3%B6rderung were detected, but none of them with any of the campaign tags attached.

Looks good to me, thank you. Sadly, I couldn't implement the closed and clicked parts but we are okay with that. I've disabled the campaign for now. Full playout will start on Monday, 4 October.

@MartinRulsch

Sadly, I couldn't implement the closed and clicked parts but we are okay with that. I

Please allow me to remind you that in your campaign's tracking concept you have planned the following comparisons:

• banner impressions for each banner type (A/B/C test)
• closing rate of each banner (A/B/C test)
• banner clicks (A/B/C test)

Without any data in the closed_by and clicked_by fields none of these comparisons will be possible, except for banner impressions which is trivial since you will know that from the way your campaign was set.

I have no idea on how to setup a campaign, but I guess someone in WMDE could give you a hand with it?
Too bad if we miss to track valuable data in this campaign if we have already managed to have them for N (N being large integer) banner campaigns since 2017 at least.

@GoranSMilovanovic I was able to add these two types. My test was successful but I guess that you want to see if it's also correctly tracked. ;) I've enabled the campaign again and let it run until tomorrow 10 am CEST. It'd be nice if you could look for new data either this evening or tomorrow morning. Thanks in advance once again!

@MartinRulsch @GoranSMilovanovic
I had a look at the event table and found entries for the 3 events in the banner:

3	WMDE_vssurvey_fall_2021_chemie	banner-closed
47	WMDE_vssurvey_fall_2021_chemie	banner-seen
4	WMDE_vssurvey_fall_2021_frau	banner-clicked
16	WMDE_vssurvey_fall_2021_frau	banner-closed
293	WMDE_vssurvey_fall_2021_frau	banner-seen
3	WMDE_vssurvey_fall_2021_lgbt	banner-clicked
3	WMDE_vssurvey_fall_2021_lgbt	banner-closed
28	WMDE_vssurvey_fall_2021_lgbt	banner-seen

There are no banner-clicked events logged for the banner in the chemistry category, but that might as well be because nobody clicked.

@kai.nissen Thank you, Kai!

I will check out everything once again in an hour or so.

@MartinRulsch @kai.nissen

Sorry for not getting back to you on this earlier, the banners seem to work, test on 2021/10/01 shows:

                        bannername seen_by closed_by mean_close_imp clicked_by
1 1 WMDE_vssurvey_fall_2021_chemie      11         2              0         NA
2 2   WMDE_vssurvey_fall_2021_frau     105        11              0          1
3 3   WMDE_vssurvey_fall_2021_lgbt       5         2              0         NA
  mean_click_imp close_rate click_rate        day              campaign
1             NA       0.18         NA 2021-10-01 VolunteerSupport_2021
2              0       0.10       0.01 2021-10-01 VolunteerSupport_2021
3             NA       0.40         NA 2021-10-01 VolunteerSupport_2021

Just please try to remember if you have clicked on WMDE_vssurvey_fall_2021_chemie or WMDE_vssurvey_fall_2021_lgbt because we have no clicked_bydata for them.

I will check for the page views of the campaign landing page now.

@MartinRulsch

Ok, something seems to be wrong about the pageviews; could you please copy and paste here exactly what landing page are we tracking in this campaign, please?

I mean the one described in the tracking concept as:

landing page 2 (WP:FÖ) views per device (desktop, mobile, Ipad) and A/B/C test

Thank you.

@MartinRulsch @GoranSMilovanovic I assume, the click handling event (which triggers logging it in Event Logging) gets lost when users are directed to a different domain. As far as I remember, the survey software supports detailed statistics that include the numbers of users hitting the survey. Wouldn't that be sufficient?

@MartinRulsch

Did the campaign start?

Also, from from T291562#7395390:

could you please copy and paste here exactly what landing page are we tracking in this campaign, please?

Thank you.

@GoranSMilovanovic @kai.nissen I'm not surprised by the low number of clicks on the second landing page. It's just mentioned at the last page of the survey and there is not much urge to click that button but a nice plus for those who seek more information about it.

Nevertheless, I browsed to https://de.wikipedia.org/wiki/Wikipedia:Förderung?campaign=WMDE_vssurvey_fall_2021_frau right now (something which I didn't do so far as I assumed that anybody else would do so during the test). It'd be nice if you could check tomorrow if this (or others https://de.wikipedia.org/wiki/Wikipedia:Förderung?campaign=WMDE_vssurvey_fall_2021_lgbt https://de.wikipedia.org/wiki/Wikipedia:Förderung?campaign=WMDE_vssurvey_fall_2021_chemie ) are reported or not. As Kai said, these exact numbers would be nice to know (no matter how low they are) but are not key to the whole campaign as it is still a test for us this year.

I've enabled the campaign and banner at 100 % now (I wanted to be around when it starts) so that we should see some larger numbers by tomorrow. Thank you both for your time and patience! I look forward to the tracking results as well as the survey responses. :)

@MartinRulsch Thank you, Martin. I am on it: following the numbers, reporting back here as of tomorrow. Good luck!

@GoranSMilovanovic As the test is over now, should this ticket be closed and the conversation continued on the main ticket [[T291547]]? I could also create an analytics sub-ticket if need be.

@MartinRulsch Please leave the ticket open until I share the first "official" dataset with you. Thank you. I will share the data very soon.

@MartinRulsch

Here are the 2020/10/04 data for banner interactions:

Please let me know if the data sense to you.

Pageviews data following soon.

@MartinRulsch

Here are the 2021/10/04 pageviews data:

From the pageviews data we can see only 3 page views of the landing page all tagged by WMDE_vssurvey_fall_2021_frau.

Let me know if this data make sense.

If both this and the dataset in T291562#7403635 are fine with you - we are putting the analytics code on a regular daily schedule and I will let you in which public directory to look for the data.

Thank you for your patience.

@GoranSMilovanovic Thank you for the data! Both files are fine with me. Please proceed with the analytical part as suggested. :)

@MartinRulsch

The aggregated data are found in the campaign public data directory.

The update will run every day at 06:00 UTC until October 17 2021.

@MartinRulsch The data acquisition for this campaign is finished now; the daily updated is stopped.

The data are here: campaign public directory.

@GoranSMilovanovic Thank you, for your kind support. :) Looking forward to working with you again (and the invoice ;D).

@MartinRulsch : ) Looking forward to working with you too :D