What definition of a Guided Tour completion do we want to use:

"Some people of our team clicked on every banner and I created some test accounts that you should be able to track. Unfortunately there was no Guided Tour shown after the registration."

Please find the test report attached.

Could you please check whether you are tracking the referer for the Registration Page (Spezial:Benutzerkonto_anlegen) when the source is /wiki/Wikipedia:Wikimedia_Deutschland/Mach_mit?

@Stefan_Schneider_WMDE No worries about the banner clicks and landing page/registration page views; I've already tracked the test data and I'm working on the report right now. Ok, so no banner impressions in the test report.

None of the following banners:

Once again, and very important:

@Stefan_Schneider_WMDE Ok, please let me know when you have some test data in; the R test code is ready. Thanks.

@kai.nissen Kai, if a user sees a General Invitation Banner and goes to the landing page

@Stefan_Schneider_WMDE What elements of the Campaign will the test encompass:

Mhmhmmm... Just beautiful. Thank you very much. As of this moment, I will start adjusting the Summer Banner Campaign R code for the Autumn Banner Campaign. That shouldn't take long. Then we test.

You're welcome @Stefan_Schneider_WMDE

@Stefan_Schneider_WMDE Simulation updated [[ https://docs.google.com/presentation/d/18Hq7ULkWVoQtMv5EI2JPcYEdIlTc2h8bgdMQwC35gE8/edit#slide=id.g2688085230_0_15 | [slide] ]].

@Stefan_Schneider_WMDE Ok, I see: if we use banner bucketing, we can have only two different banner (please confirm).

@Stefan_Schneider_WMDE No no no please use banner bucketing, now I understand what it does. Certainly there is a way to handle the situation where the user sees two or more banners during the campaign analytically, but I can already sense complication there.

@Stefan_Schneider_WMDE One thing only, please:

@Stefan_Schneider_WMDE Of course. Stay in touch.

Obviously, that does not correspond to the Campaign design as provided in the KickOff new editor autumn edition slides.

Why would anyone need to search for banners in the page views?

@Addshore Thanks!

Let me summarize the whole tracking schema for the Autumn Banner Campaign 2017 for us and ask @Addshore for a review and confirmation/correction:

Please implement the following user tracking schema for this campaign (these data should end up in ServerSideAccountCreation_5487345 for campaign tracking)

@Addshore Here we go:

@Addshore Thank you for this thorough inquiry. Yes, the normalized vs. raw request count difference is a reflection of the sampling rate, I can't see any other explanation (besides, the numbers fits nicely; for our Autumn campaign, we will use only the Webrequest table, no Pivot, and normalize the counts accordingly in R).

@Verena @Addshore Do not forget about the MW parameter that needs to be set in order to make the tracking of the Guided Tour data map into the MariaDB SQL tables (the one we've discovered sometime during the Summer Banner Campaign).

• Pivot goes out of the loop; data from the Webrequest table (Hadoop) match perfectly with what Pivot reports.

@JAllemandou @Addshore Here's the description of the discrepancy found during the WMDE Summer Banner Campaign 2017:

@JAllemandou Thank you very much.

@JAllemandou That's right. I'm puzzled: today, I was able to find only one WMDE campaign (some test thing) there.

@Addshore Reviewed, +1. Please wait until the next step for us to get in touch (Hangouts, Data Analysis Weekly, something) first.

@JAllemandou Is that what prevents me to find any wdmesummer campaign when I try to filter by campaign in Pivot?

@JAllemandou Thanks. I've already read the Oozie page. As @Addshore would put it, in theory I got it, doing it is most probably walking a quite different pair of shoes.

@Addshore Understood: so, the Oozie coordinators are actually responsible for setting the script parameters like destination directories and similar. Everything seems clear here except for the

@JAllemandou Also, I am interested to learn about the data retention lifetime in Druid. For example, the WMDE Summer Banner Campaign 2017 data (wmdeSummer1, wmdeSummer2) cannot be accessed through Pivot anymore. Unfortunately, the Druid/Pivot documentation on Wikitech (Druid page, Pivot page) does not enlist this.

@JAllemandou In the HiveQL script that updates Druid, the following line:

@Tobi_WMDE_SW Please review, looks like we're done here.

Status report:

• All requested singlestats are now present.

@Addshore Ok. Percent Conflicts Resolved singlestat is wrong, in spite of the fact the ratio is correct - check it out.

@JAllemandou Thanks.

Also - now this is really strange - when we have three queries defined like in the previous example:

No change in behavior when two series are defined and the difference is computed in the third query:

Using the diffSeries() function, as described in the Graphite documentation on: http://graphite.readthedocs.io/en/latest/functions.html:

@Addshore @Tobi_WMDE_SW Well, this why I've asked on how to develop a better understanding of the Graphite/Grafana data structures and the way they work:

Here's the final report on the Summer Banner Campaign 2017. Section 6. covers campaign analytics. Please skip over the technical parts of the ou{F9253916}tput and focus on my explanations of the results in the narrative. Please contact me with any questions that you might have after you have read the Section 6.

• Fixed table widths under Chromium; that most probably does the job on Chrome too;
• Please ask someone to test the WDCM Overview Dashboard in Safari; again, its on: http://wdcm.wmflabs.org/WDCM_OverviewDashboard/

1. The Tables

• User registration data collected

Merged with https://phabricator.wikimedia.org/T171420: Migrate New Editors Analytics to Production on 08/29/2017.

@Verena Thanks. I'm on it.

@Verena Which sheet for the complete analysis? And what editing behavior do you mean exactly - edits of new users (after the campaign, as indicated in No. 2 on the list below)?

Current status: solving the R {sparklyr} connectivity to cluster issue in production, see: https://phabricator.wikimedia.org/T139487

@mpopov I've tried out two different approaches to connect to Spark from {sparklyr} on stat1005, including yours, and failed. Please take a look and let me know if you have a clue on what's going here. Thanks a lot.

@Verena As soon as https://phabricator.wikimedia.org/T171049 is closed, I'm on this one.

@Verena The user registration timestamps and the data on whether the Guided Tour was shown or not can be matched.

Here's the data on Guided Tours from the Summer Banner Campaign.

• As of what could have been done for the WDCM in the Map-Reduce framework, this is it. All scripts checked, all in use. Closing the task.

@Addshore Thanks. I am still experiencing problems with Gerrit (for example, "git review -s" seems to be deprecated in favor of doing something else), but I don't think it will cause me too much trouble in the long run.

@Addshore Thanks for picking this up!

The request on https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests is finally processed, there is a Gerrit repo for WDCM now:

• Pre-processed tables for the new WDCM_Process.R (analytics, essentially) module are ready.

@Addshore Thanks, I'll take a look, however we still don't have a Gerrit repo for WDCM: my request on https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests is there several days already.

@Verena @Addshore Nothing yet in relation to https://phabricator.wikimedia.org/T171958

Since stat1002 is replaced by stat1005, the WDCM Process Module will be deployed on the new statsbox.

A quite detailed draft: Wikidata Concepts Monitor (WDCM) - Technical and Data Documentation + System Design is shared on:

@Lydia_Pintscher @daniel Please place your final comments on https://docs.google.com/document/d/1QfspTewm9SdWfuB0EBFAMDBW8C4FP7qeF_V43e53frE/edit and let's close this debate. Thank you.

@Lydia_Pintscher Great! Closing the task.

Hey, take a look at http://wdcm.wmflabs.org/WDCM_Dashboard/

@Lydia_Pintscher Slides here.

@Lydia_Pintscher Let me know how do these look to you. First, try to interpret what you see without any guidance that I could possible provide. Then, it would be good to have a short Google Hangout and see up to what degree do we understand each other without additional explanations. That would be important because similar things will be entering the WDCM Dashboards. Finally, I will send you these visualizations in a slide deck for your Wikimania presentation, each visualization accompanied by a short explanatory narrative.

I would help, but ...

wdcm_maintable, the Hive table from which all further pre-processing for WDCM is done, is successfully created. Approx. 12 billions rows are there.

@Addshore Thanks Adam.

@Addshore What exactly do I need access to - could you please provide an answer to this question since I still haven't seen the inside of this server? Thanks.

@Verena @Addshore Correct; I have requested access to the log server here https://phabricator.wikimedia.org/T171958, and it might take up to three working days to get it - or at least the Ops say so.

Ok, let's get analyzed what we can have analyzed from the Summer Banner Campaign.

The first test scripts were run during the last night.

@Addshore As of the Guided Tours logging, please take a look at the following discussion: