Page MenuHomePhabricator

Deliver Daily Reports and the Final Campaign Report
Closed, ResolvedPublic

Description

Please provide daily reports during the campaign week: 6.-12.5.2019

Please provide two weekly reports (with daily data) two weeks afterwards: 13. - 26.2019

Here are the specifics of the data needed: [doc will be linked asap]

Event Timeline

@Stefan_Schneider_WMDE

I will use a Drive Document to share daily reports.

@Stefan_Schneider_WMDE @Verena @kai.nissen and @RazShuty currently have access. Please let me know if I need to include anyone else there. Thanks.

@GoranSMilovanovic: Regarding the document and the first data I have a quite urgent question. Do we need another extension/link to the buttons of the first landingpage to track how much users were on the second landingpage? I cannot decifer the data in the doc to get that number. Can you help me here?

@Stefan_Schneider_WMDE If you mean the https://de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia page: I don't think we need any additional tracking, I simply don't have it in my analytics code yet (and thanks for noticing this)!

Let me re-run the analytical code again, explicitly including this page now, and I will get back to you then.

@Stefan_Schneider_WMDE I am afraid that you will have to add some tracking after all:

uri_queryuri_pathCountCampaignDay
/wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia2682018-08-01
?campaign=WMDE_2018_sprbt1/wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia52018-08-01
?campaign=WMDE_2018_sprbt2/wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia92018-08-01
?campaign=wmde_etc2017_bt1/wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia12018-08-01
?campaign=WMDE_neweditors_summer2018_1/wiki/Wikipedia:Wikimedia_Deutschland/Neue_Ehrenamtliche/FAQ61722018-08-01
?campaign=WMDE_neweditors_summer2018_1%23Wer-darf-schreiben/wiki/Wikipedia:Wikimedia_Deutschland/Neue_Ehrenamtliche/FAQ22018-08-01
?campaign=WMDE_neweditors_summer2018_2/wiki/Wikipedia:Wikimedia_Deutschland/Neue_Ehrenamtliche/FAQ76692018-08-01
?campaign=WMDE_neweditors_summer2018_2%23Wikipedia-Buero/wiki/Wikipedia:Wikimedia_Deutschland/Neue_Ehrenamtliche/FAQ12018-08-01
?campaign=WMDE_neweditors_summer2018_3/wiki/Wikipedia:Wikimedia_Deutschland/Neue_Ehrenamtliche/FAQ57262018-08-01
?campaign=WMDE_neweditors_summer2018_3&veaction=editsource/wiki/Wikipedia:Wikimedia_Deutschland/Neue_Ehrenamtliche/FAQ12018-08-01
?campaign=WMDE_neweditors_summer2018_3%23Schutz-vor-Missbrauch/wiki/Wikipedia:Wikimedia_Deutschland/Neue_Ehrenamtliche/FAQ12018-08-01

As you can see from the table (I've run the test for 1. Aug 2018 only, should be enough), the /wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia is never referenced by any of the campaign tags that we maintain now (c.f. the uri_query column).

The HiveQL query used to test intentionally avoids to select the uri_query parameter for the /wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia page in order to allow us to see where is it referenced from at the current moment. As the table shows, this page receives hits from ?campaign=WMDE_2018_sprbt1, ?campaign=WMDE_2018_sprbt2, and ?campaign=wmde_ect2017_bt1 still, but nothing from the actual campaign.

USE wmf;
  SELECT uri_path, uri_query, referer FROM webrequest
  WHERE uri_host = 'de.wikipedia.org'
  AND ((uri_path = '/wiki/Wikipedia:Wikimedia_Deutschland/Neue_Ehrenamtliche/FAQ' 
 AND uri_query LIKE '%campaign=WMDE_neweditors_summer2018%')
  OR (uri_path = '/wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia')) 
  AND ((year = 2018 AND month = 08 AND day = 0 AND (hour = 22 OR hour = 23)) OR (year = 2018 AND month = 08 AND day = 01 AND (hour >= 0 OR hour < 22)))

I suspect the first row of the table, where the uri_query parameter is missing, encompasses the data from our current campaign too.

@GoranSMilovanovic OK. That's a severe problem. I thought, that the extension in the banner is stored in a coockie and is still there when they get to the next landingpage? Now I'm wondering witch extension I can add to the buttons in the first landingpage to ensure, that we can still track them regarding the banner they initially clicked...

@Addshore We have an URGENT tracking issue. Can you answer the questions above?

Just to be clear: The tracking information you can find here: T199689

@GoranSMilovanovic I have another question to the data-table: Are you sure, that we have only around 8.000 impressions per day? Usually it is around 1.800.000 impressions per day, when we go with 80% banner diet. Could you check that? I want to be sure, if the banner setup is right.

@GoranSMilovanovic I now (15:10 CEST) corrected the Links in the buttons on the first landing page and added another extension: ?campaign=WMDE_neweditors_summer2018_LP2 Let's at least track how many people came from the first landingpage.

@Stefan_Schneider_WMDE Hey: if your new tracking of the /wiki/Wikipedia:Wikimedia_Deutschland/LerneWikipedia page is tagged by the WMDE_neweditors_summer2018_LP2 campaing tag - yes, I can see it now (from incomplete data on 03/Aug/2018). Please confirm.

@Stefan_Schneider_WMDE Daily reporting is now updated.

NOTE: Daily Banner Impressions, Daily Pageviews, and Daily Registrations are provided on separate sheets now.

@GoranSMilovanovic I have another question to the data-table: Are you sure, that we have only around 8.000 impressions per day? Usually it is around 1.800.000 impressions per day, when we go with 80% banner diet. Could you check that? I want to be sure, if the banner setup is right.

@GoranSMilovanovic Hi Goran, I guess you've missed that due to the other urgent issues. But could you please double check the impression rates? I'm still a bit confused.

@Stefan_Schneider_WMDE Checking now and getting back to you as soon as it's done.

@Stefan_Schneider_WMDE From what I see this is constant: recordImpressionSampleRate=0.01. Please confirm that the recordImpressionSampleRate of 0.01 does not change across the campaign days and I will deliver a number of banner impressions corrected for the impression sample rate parameter.

@GoranSMilovanovic Hi Goran, I don't know if this parameter was the same in every campaign and I don't know where to find this information. @Tim_WMDE Do you have an idea about that parameters and can help here?

@Stefan_Schneider_WMDE Don't worry, I know where to find the information. The question is just whether the impressions sampling rate does, or does not, change during this campaign. I remember we had only one campaign where this parameter was not constant, and everything else had a constant sampling rate of .01. Also, this information should be provided before the onset of the campaign, in spite of the fact that it is always easy and possible to adjust the dataset in respect to the values that it takes.

Hey @Stefan_Schneider_WMDE, @GoranSMilovanovic is correct, only 1% of users is recorded so you have to multiply all results by 100 which seems to roughly get you 1.9 - 2.0 million banner impressions which you expected to get.

I am not aware of anyone planning on changing the sample rate value, most of the relevant people up in the third floor are on vacation anyway and as far as I know the value was not to be changed. I highly doubt the value will change unless there is another party involved that I am not aware of.

@Tim_WMDE Thanks, Tim. Adjusting the data set per banner impression sample rate right now, updating the daily report very soon.

@Stefan_Schneider_WMDE The data on banner impressions in the daily reporting table are now adjusted for impression sample rate of p = .01.

@GoranSMilovanovic @Tim_WMDE Thank you guys for prompt solution here. Now everything looks perfect!

@Stefan_Schneider_WMDE Daily update for 8. Aug is complete.

Please confirm if the campaign has ended and I will start working on the Final report in line with the shared campaign doc.

@GoranSMilovanovic I confirm, that the campaign has ended yesterday at 12:30 UTC. We would be more than glad to see the complete data and see what we have reached with our campaign.

@Stefan_Schneider_WMDE The final campaign report will be delivered as soon as it is completed and certainly until tomorrow.

GoranSMilovanovic renamed this task from deliver daily reports to Deliver Daily Reports and the Final Campaign Report.Aug 9 2018, 11:46 AM

@Stefan_Schneider_WMDE Here is the campaign Final Report v1.0 (no training module data yet, banner events are still not analyzed, no specific research questions are addressed; however, most of the essential overviews that I give you for every campaign are already included in this version):

My notes and questions to you:

  • 1.2 Banner Clicks (Pageviews). I am not sure if I understand the difference between the various sources of pageviews (i.e. banner clicks) here. For example, we have:
    • WMDE_neweditors_summer2018_1 and WMDE_neweditors_summer2018_1%23Wer-darf-schreiben
    • WMDE_neweditors_summer2018_2 and WMDE_neweditors_summer2018_2%23Wikipedia-Buero
    • WMDE_neweditors_summer2018_3 and WMDE_neweditors_summer2018_3%23Schutz-vor-Missbrauch - I understand that each banner pointed towards a specific anchor on the landing page, but if the same banner, for example `WMDE_neweditors_summer2018_1 has always pointed towards the same anchor (related to %23Wer-darf-schreiben in this example, presumably) then what is difference in registering a pageview from WMDE_neweditors_summer2018_1 and WMDE_neweditors_summer2018_1%23Wer-darf-schreiben?
  • 1.2 Banner Clicks (Pageviews).
    • What do these two represent: WMDE_neweditors_summer2018_3&veaction=editsource and WMDE_neweditors_summer2018_2&ios-app-redirect=true?
  • 1.2 Banner Clicks (Pageviews).
    • The No referer source probably has to do with the tracking later (on 03. August 2018, more precisely) implemented as WMDE_neweditors_summer2018_LP2; however, it is present during the whole course of the campaign, only with a significant downfall following 04. August.

I would be thankful If you could help me clarify the meaning of the above discussed sources of pageviews in this Campaign. Thank you.

Also, please note that the user edits categories that you have requested this time (1 edit, 2 - 4 edits, 5 - 10 edits, >10 edits) are not the same as we have used earlier. So, if you need this data for reasons of comparison, maybe we should decide to standardize the categories of user edit observations.

Finally, please let me know how would you like to analyze the banner events dataset, i.e. what is the essential information that we are looking for there. Thanks!

@GoranSMilovanovic Thank you for the report.

Regarding your questions:

  • I was confused too when I saw the first data set. The questions are commonly present in my head and I hoped, that you can give an answer to them :P. I guess, that there is no difference between the anchor-tag (eg. WMDE_neweditors_summer2018_1%23Wer-darf-schreiben) and a pure tag (eg. WMDE_neweditors_summer2018_1). So I would suggest to put them together. It's only a few, but I cannot explain why there are two campaign tags.
  • What do these two represent: WMDE_neweditors_summer2018_3&veaction=editsource and WMDE_neweditors_summer2018_2&ios-app-redirect=true? I cannot answer that question. @Tim_WMDE Do you have an idea what this could mean?
  • I guess the no referer source could refer to the third link in the first landing page. I just noticed that the third link to the second landingpage was not marked with the campaign tag WMDE_neweditors_summer2018_LP2. I think then we just have to take the page views in general. because noticing that the data is corrupt and we also have only data from the 03. August. The total page views from the second landing page are also not really trackable down to klicks on the first landingpage, but this button was the only way to get to the second landingpage (besides our link on our wikipage about the campaign. I guess there are not so much clicks during the campaign). What source would you suggest to look at?

Thank you for adding this section (1 edit, 2 - 4 edits, 5 - 10 edits, >10 edits). I normally took this data from this table. So it was kind of always there. But we can add this in general because I noted it anyway.

Bildschirmfoto 2018-08-10 um 13.33.50.png (884×1 px, 253 KB)

To your last point: I would like to know which banner worked best in terms of registration and the conversion to first and second landing page. Of course the second lang pages conversion we cannot measure. Do you have an idea how to manage that?

Ans finally another point: We would like to see how the user edits are changing one week after the campaign and two weeks after the campaign. Could you provide such a graph (as posted above) for one week (15.08) and two weeks (22.08) after the campaign ended?

@Stefan_Schneider_WMDE

What do these two represent: WMDE_neweditors_summer2018_3&veaction=editsource and WMDE_neweditors_summer2018_2&ios-app-redirect=true? I cannot answer that question. @Tim_WMDE Do you have an idea what this could mean?

These users probably clicked on the banner "under some weird conditions" or purposefully modified the URL. The first one is related to someone trying to use VisualEditor on the page and the second one seems to be someone using an Apple device while having the official Wikipedia app installed (since mobile devices were excluded I am not sure how they got the banner like that, I imagine someone was doing some "hacky" stuff).

I had a quick glance at the data and it seems like these are just single cases (1 and 2 users respectively) so you can ignore / merge those numbers.

@Stefan_Schneider_WMDE Here goes a corrected and update (w. training modules data) final report for the Summer Banner Campaign 2018:

NOTES

  • All variants of WMDE_neweditors_summer2018_1, WMDE_neweditors_summer2018_2, and WMDE_neweditors_summer2018_3 were merged.
  • In the first version of the report I did not separate the page views for the first and the second landing page - I apologize for that, I did not pay enough attention and didn't realize that they are present in the same data set (see Chart 1. 2. Daily Banner Clicks (Landing Page Views) in the current version of the report).
  • The training modules analytics are now included @Ragesoss thank you for the data set.

@Tim_WMDE Thank you for clarifying the "odd cases" in T199694#4494124.

@Stefan_Schneider_WMDE

Ans finally another point: We would like to see how the user edits are changing one week after the campaign and two weeks after the campaign. Could you provide such a graph (as posted above) for one week (15.08) and two weeks (22.08) after the campaign ended?

We will have this.

To your last point: I would like to know which banner worked best in terms of registration and the conversion to first and second landing page. Of course the second lang pages conversion we cannot measure. Do you have an idea how to manage that?

Let me take a look and I will get back to you on this.

@Stefan_Schneider_WMDE Here goes a corrected and update (w. training modules data) final report for the Summer Banner Campaign 2018:

NOTES

  • All variants of WMDE_neweditors_summer2018_1, WMDE_neweditors_summer2018_2, and WMDE_neweditors_summer2018_3 were merged.
  • In the first version of the report I did not separate the page views for the first and the second landing page - I apologize for that, I did not pay enough attention and didn't realize that they are present in the same data set (see Chart 1. 2. Daily Banner Clicks (Landing Page Views) in the current version of the report).
  • The training modules analytics are now included @Ragesoss thank you for the data set.

@Tim_WMDE Thank you for clarifying the "odd cases" in T199694#4494124.

@Stefan_Schneider_WMDE In relation to the following question:

I would like to know which banner worked best in terms of registration and the conversion to first and second landing page. Of course the second lang pages conversion we cannot measure. Do you have an idea how to manage that?

As of the conversion towards the second landing page, well, we do have some data in our spreadsheet now, but I really hope you will be able to clarify their meaning to me. I have cross-tabulated the uri_query and the referer field for all pagevies of the second (i.e. LerneWikipedia) landing page. Please go and take a look.

@GoranSMilovanovic Thank you for the updated report! I checked the training module data and unfortunately not many people used the training modules, so guess this is not really enough data to make some guesses about the modules and the answers to our initial questions.

Great that we can have data after one and two weeks. After we have that, we would look at the data with our team again. If it is sufficient for you you could just add the data in the overall report after that 2 weeks – so on 23.08 – and post it again?

The data regarding the second landing page we can discuss in our spreadsheet. Looking forward to finding a solution for that!

@Stefan_Schneider_WMDE We will have the update on user edits after (a) one week, posted here tomorrow, and (b) two weeks, posted here next week. The banner closing quote data will be included in the tomorrow's update.

Hi Goran, great! Then we will du it that way. Could you update the closing
rates in that table please?
https://docs.google.com/spreadsheets/d/1E36H-QCS7VD8H5SId5wcS4xp_yGhVnIUoQ1UxGEwux4

GoranSMilovanovic <no-reply@phabricator.wikimedia.org> schrieb am Mi. 15.
Aug. 2018 um 12:17:

GoranSMilovanovic added a comment.

@Stefan_Schneider_WMDE
https://phabricator.wikimedia.org/p/Stefan_Schneider_WMDE/ We will have
the update on user edits after (a) one week, posted here tomorrow, and (b)
two weeks, posted here next week. The banner closing quote data will be
included in the tomorrow's update.

*TASK DETAIL*
https://phabricator.wikimedia.org/T199694

*EMAIL PREFERENCES*
https://phabricator.wikimedia.org/settings/panel/emailpreferences/

*To: *GoranSMilovanovic
*Cc: *Ragesoss, Tim_WMDE, Addshore, kai.nissen, Verena, RazShuty,
GoranSMilovanovic, Stefan_Schneider_WMDE, sonkiki, Cirdan

@Stefan_Schneider_WMDE Banner closing clicks and the respective percents (vs. impressions) are now in your reporting table.

Let' stay in touch in relation the pageviews for LP2 and see if we can think of something that would help us reconstruct the data (if possible at all given the constraints imposed by the campaign tracking setup).

Tomorrow: user edits after one week, and next week: user edits after two weeks.

@Stefan_Schneider_WMDE As already mentioned in my recent e-mail to you, there were no new edits on behalf of those users who have registered with our Summer Banner Campaign 2018. I was a bit surprise to learn about that so I have double checked the whole procedure that collects user edits, and no, there were no new edits.

@GoranSMilovanovic Oh, that's a bummer. Then let's see in one week, if there are more edits. Thanks for checking the whole procedure again!

@Stefan_Schneider_WMDE Once again: I didn't believe my eyes at the first glance and I have then double checked the code that is reliable and running for several WMDE campaigns already. No way the procedure could have got it wrong: step 1, fetch all IDs who have registered with the campaign, step 2, collect everything from the dewiki.revision that was registered under that list of IDs. As you say: let's talk in one week from now.

@Stefan_Schneider_WMDE Two weeks following the end of the Summer Banner Campaign 2018: still no user edits.

@GoranSMilovanovic Thank you for keeping up. Quite unfortunate that the numbers didn't change.. I'll get back to you in the upcoming days regarding the other numbers. I'm quite full right now.