@Janina_Ottma_WMDE The 2020/01/16 update is complete; two new user registrations.

@Janina_Ottma_WMDE Running the 2020/01/16 update now.

How are these edits split by edit class (as in final report 3.1.)? Could you add this information?

Also if you say the numbers in the spreadsheet are the ones to trust, we would need the timeframe from October 28th onwards because the newsletter was send out on October 28. Could you add them to the table?

So just to be sure: these are all page views of Wikipedia_vor_Ort in November from the tag WMDE_neweditors_autumn_2019_nl_lp1?

In reference to: T235839#5809630

In reference to: T235839#5809565

@Christine_Domgoergen_WMDE I will try to respond to T235839#5809565 and T235839#5809630 tonight, but I can do it only later tonight.
I will email you if I make it just to make sure you get the data in the morning at least.

@Christine_Domgoergen_WMDE And here's the November 2019 pageviews data for the WMDE_neweditors_autumn_2019_nl_lp1 newsletter banner:

@Christine_Domgoergen_WMDE We have the October 2019 data (or what is left of them because of the wmf.webrequest purge after 90 days):

@Christine_Domgoergen_WMDE I think I will be able to deliver the pageviews until this evening, and probably even earlier, say around 18:00 CET.

@Christine_Domgoergen_WMDE The pageviews thing will take a while, sorry - it's just that the code needs to search through a lot of data and was hitting heavy against the cluster resources, so I had to switch it to search day by day through October and November 2019. I will be reporting back as soon as I have something.

@Christine_Domgoergen_WMDE In order to compare the results, we will pick up everything on:

Page Views Wikipedia_vor_Ort and LerneWikipedia: in the page view tool of wikipages we have quite some different numbers (see links), which of course differ a bit but on some days are far higher than the page views you found in the database. Can you check again, if the numbers in the report are correct?

@Christine_Domgoergen_WMDE As of the following:

In an early comment in the phab ticket you mentioned some user registrations happening before the start of the banner campaign (28. - 31.10.19) via the newsletter tags (WMDE_neweditors_autumn_2019_nl_lp2 and WMDE_neweditors_autumn_2019_nl_lp1). In the final report those registrations do not show up again, maybe because the time span of the report starts at the 1st of November? Could you have a look at it again and if you can confirm those registrations update the final report and the spreadsheet if necessary? The campaign time frame starts not only with the banner on Nov 1st but with the Flyers on October 7th (see dates in the graphic in the tracking doc). Could you double check again, that we have all the figures from this whole time span included in the report? This is also relevant for the page views coming from flyers for example. Thank you!

@Christine_Domgoergen_WMDE The ticket is re-opened in respect to the following requests:

@Janina_Ottma_WMDE The 2020/01/14 and 2020/01/15 updates are ready, no new user registrations.

@Janina_Ottma_WMDE The 2020/01/13 update is ready, no new user registrations.

The only thing that I do not understand here is the following planned column:

@Janina_Ottma_WMDE @tmletzko Thank you!

@Janina_Ottma_WMDE 2020/01/11 update is ready.

Update for 2020/01/10 is in the Spreadsheet, one fresh user registration on January 10.
@Janina_Ottma_WMDE Please let me know until when do we run this campaign. Thank you!

Updates for 2020/01/07 and 2020/01/08 included, no new user registrations since January 5.
@Janina_Ottma_WMDE When does this campaign end?

Updates for 2020/01/05, 2020/01/06, and 2020/01/07 are now included.

Update for 2020/01/04 is ready.
No new user registrations on 4. January.

Update for 2020/01/03 is ready.
No new user registrations on 3. January.

Wikidata Pageviews per Namespace Dashboard is finally operational again.

Update for 2020/01/02 is ready.

The Daily Reporting Spreadsheet for this campaign is ready.

@kai.nissen There were six (6) user registrations from the following banners on 2019/12/30:

Wikidata Pageviews per Namespace Dashboard not responding following the changes in T239199 (Kerberos Auth for all WMDE Analytics):

@WMDE-leszek Please consider this ticket and help me prioritize. Thanks.
@A00604591 Please let me know if there is a specific time frame for this.

The analytics code for this campaign is in place.

@Janina_Ottma_WMDE To put it in a nutshell: we can begin testing on Monday, December 23.

@Addshore @Jan_Dittrich Here is the summary of the approach to collect the baseline data, following our today's meeting:

@kai.nissen And you're finally back - thank you very much for the clarifications in T240361#5744403 and welcome!
@Janina_Ottma_WMDE More or less I think I get the tracking of the campaign right, let's discuss the details in our forthcoming call.

• WD_percentUsage_PRODUCTION.R is back on crontab on stat1004;
• monitoring; next steps:
• WD_PageviewsPerType_Engine.R on stat1007;
• WDCM (T)itles and WDCM (S)itelinks - HiveQL;
• all WDCM Pyspark ETL.

@Addshore No worries you've been pinged only because you were involved in this banner actions thing in the previous campaign.
Ping: @Tim_WMDE @awight Please, gentlemen - we need to see what exactly about banners we can learn and from what schema. Thank you.

Temporarily not on crontab:

@Addshore Well, now it sounds even more complicated than in the ticket description.

@Janina_Ottma_WMDE At this point, your tracking definitions are completely unclear.

@Lydia_Pintscher Finally, a "non-Commons" data quality report is ready: Wikidata Quality Report - Commons Excluded.
This one encompasses only items that are reused anywhere except in Wikimedia Commons.
Let me know if anything else needs to be done here. N.B. Statistical hypothesis testing comparing various subsets (e.g. Commons vs. Everything else) are possible.

@Lydia_Pintscher We now have a separate WD Quality Report for Wikimedia Commons.
Working on its complimentary, "non-Commons", quality assessment now.

Done.

My initial observations (continued) - please comment:

My initial observations - please comment:

• Both dashboards repaired;
• The problem, however, is not so obvious:
  • some data set URLs where not properly url encoded, yet
  • the dashboard was responsive in spite of that fact just until recently.

Do we need any additional work on this or shall we close the ticket? Thanks.

Once again, please let me know if you need anything else on this campaign or we can close this ticket. Thank you.

in relation to your question on can we learn what external pages point to some pages in the Wikimedia universe (e.g. the target page of your research project - https://en.wikipedia.org/wiki/Hurricane_Hazel), it seems that the WMF has access to the Google Console service:

this is my first time working with Phabricator, so sorry in advance for any future mistakes - I am still learning

No worries, I will help in the process.

@Lydia_Pintscher The WD Data Quality Report is now updated.

@Janina_Ottma_WMDE If and when you update the tracking info for this campaign, please do not forget to update the description of T240351. Otherwise we might run into confusion. Thank you.

Please let me know if anything else is needed here or we can close this ticket.

Upon another week of observation (2019/12/01 - 2019/12/07) we find no new data for this follow-up.

Since @JAllemandou has kindly provided a fresh copy of the WD JSON dump in hdfs (T209655#5713452), our next WD Quality Report update will be constrained only by the timestamp of the most recent snapshot of the mediawiki_history available to us.

@JAllemandou Thank you - as ever!

I care about the storage issues. I dropped unneeded lines and freed 70GB in stat1007.

@Ladsgroup As of me, you don't need to keep the following ORES updates on stat1007 anymore:

@JAllemandou Do you think it would be possible to produce a new version of this data set?
The latest update seems to be: 2019-10-03 09:29 /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20190902 - which you have pointed me at in T209655#5543575.
I would need to update the Wikidata Quality Report soon (Dec 15, say), and the code relies on Spark to process the dump. Thanks.

I have started a gradual update procedure (fromt the initial 111Gb run.out -> run_20190901.out -> run_201910.out -> run_201911.out).

• Data sets are now complete following a re-run of the wdcmModule_Orchestra.R;
• Next steps: figure out {curl} (possibly) related problems on WDCM Semantics and WDCM Geo.

@Ladsgroup Got it.

@Ladsgroup Pong. I am on it as soon as I figure out what is wrong with this thing in WDCM: T239196.

• WDCM_Sqoop_Clients.R unexpectedly took ~13h to update;
• wdcmModule_Orchestra.R was thus run (on 10:00 UTC) out of sync with the wdcm_clients_wb_entity_usage table;
• and this is the possible cause of the observed missing data.

• Moreover, dewiki and potentially other wikies are not found in the 2019-12-01 11:44 update at all;
• However, dewiki was sqooped by WDCM_Sqoop_Clients.R from stat1004:

WDCM public data set:

• wdcm_project.csv
• Update timestamp: 2019-12-01 11:44

has no data on dewiki and many other.
Inspecting now.

• This is more serious than what it seemed to be following my initial assessments.
• Possible cause: change in R packages used on the dashboard, possibly {curl}.
• Inspecting now.

The new Follow up tab in the Email Campaign Spreadsheet has the pageviews for this follow-up.

As of T235839#5691491: my bad, the data were sparse and I simply didn't spot the tags, they are in place with https://de.wikipedia.org/wiki/Wikipedia:Oberlausitz/Wikipedia_vor_Ort_2019 as in any other campaign page - check the campaign spreadsheet please.

While we wait for the public data review in T239393 to complete, here are the additional data sets.