@Lydia_Pintscher

That's "our" information. And then we have links/external identifiers to say 3 libraries that also have information about X. We want to somehow quantify the latter for all of Wikidata's entities.

Q. Do I understand correctly: you would like to have some sort of comparison (a "ratio" of some form) between (a) knowledge on X in Wikidata and (b) knowledge on X in other (linked from Wikidata) databases?

We should try to find ways to quantify this information.

@Lea_WMDE You can now test your new dashboard: http://wmdeanalytics.wmflabs.org/WD_pageviewsPerNamespace/

• data set review requested from Analytics in T227905;
• next steps:
  • visualizations + dashboard;
  • test, deploy.

• data set production - completed.

@Lea_WMDE I guess 640 is the EntitySchema namespace (figured this out from this Gerrit patch, since it is not documented in the Wikidata namespaces), right?

• Deployed everything.

@Christine_Domgoergen_WMDE Then please close this ticket and open a new one when the campaign is ready. It will take me no time to set up the analytics for the new campaign if only parameters like landing pages are different. Thank you.

@Christine_Domgoergen_WMDE Will we be using exactly the same parameters as in this campaign, or are we talking about a new, different campaign altogether?

2019/06/21: no pageviews, no user registrations.
2019/06/22: no pageviews, no user registrations.
2019/06/23: no pageviews, no user registrations.
2019/06/24: no pageviews, no user registrations.
2019/06/25: no pageviews, no user registrations.
2019/06/26: no pageviews, no user registrations.
2019/06/27: no pageviews, no user registrations.

@bd808

2019/06/14: pageviews: staa = 2, strz = 1 , user registrations: 0.
2019/06/15: pageviews: strz = 1 , user registrations: 0.
2019/06/16: pageviews: none, user registrations: 0.
2019/06/17: pageviews: none, user registrations: 0.
2019/06/18: pageviews: none, user registrations: 0.
2019/06/19: pageviews: strz = 1, user registrations: 0.
2019/06/19: pageviews: strz = 1, user registrations: 0.

@Lea_WMDE Do we have any additional requirements here or shall we resolve the ticket?

• The only two dashboards that now depend upon WDCM_updateLabs.R are

• Refactored.

• wdcm_maintable removed from hdfs;
• all WDCM dashboards now running Apache Spark supported update engines;
• resolved.

The WDCM Biases dashboard is now back on updates:

• it will be update once monthly, and
• it's update is dependent upon the most recent version of the WD dump copy in hdfs (see T209655);
• the dashboard is now fully client-side dependent,
• and because it now uses Spark as its ETL back-end we can finally remove
• the huge wdcm_maintable from HDFS that previously supported our ETL procedures for the WDCM system (Hive; see T214586).

• Goodbye Advanced Search Extension dashboard :(

• archive the Advanced Search Extension dashboard (tracking).

@JAllemandou Thanks for the recent 20190603 dump copy in HDFS.

• Pyspark ETL procedures for WDCM Biases completed;
• next step: re-factor R code to work with Spark outputs.

• The WDCM Biases Dashboard's back-end has to switch to Apache Spark;
• it maintains large data sets, depends upon WDQS heavily by sending queries that time-out every now and then; beyond that,
• it relies on geo-coordinates which, when fetched together with items from WDQS, produce long vectors that cannot be converted from raw to character() in R (and which then need to be parsed by {jsonlite} into readable data.frame types for processing).

• Next step: completely remove WDCM_updateLabs.R from the WDCM pipeline.

• WDCM_Process.R is now deprecated.

• The dashboard is now fully client-side dependent;
• update_labs module - hopefully to be fully deprecated soon - updated.

@elukey I was able to reduce the amount of disk usage in my home directory on stat1007 by approx. 40Gb.
My apologies: there were some old, raw data sets there that I have forgot to clean up...

@elukey I'm on it.

Of course. Please state exactly (FROM - TO) the dates for the remaining weekly reports. Thanks.

• The WDCM Biases Dashboard is the only remaining dashboard whose back-end relies on this Hive table.
• As soon as the changes are implemented there, the table will be removed from hdfs.

@Christine_Domgoergen_WMDE Can we close this ticket now?

• All changes implemented: {text2vec} procedures replacing {maptpx};
• Deploying now; the test run was successful so the dashboards are already update from the new ML back-end;
• first run in production scheduled for tomorrow, June 7/2019;
• behavior: too many topics in the optimal model from perplexity based model-selection;
• action: we will switch the WDCM model selection procedures from perplexity based to coherence based (as in WDCM Sitelinks and Titles).

• WDCM Geo is back on updates, re-designed, fully client-side dependent, and does not rely on the wdcm_maintable anymore (see T214586):

http://wmdeanalytics.wmflabs.org/WDCM_GeoDashboard/

• WDCM Geo Dashboard does not depend upon this table anymore.

• WD_percentUsageDashboard update time changed: from 08:00 UTC to 12:00 UTC.

@Lydia_Pintscher I got it:

• Correction. It is not the Apache Sqoop procedure in WDCM_Sqoop_Clients.R that fails (luckily!);
• explanation: I've stumbled into an unfinished log file when I was checking this... and some databases, naturally, where missing - because the procedure did not complete all the passes yet.
• Continue with: inspect why the usage number reported on WD_percentUsageDashboard 'oscilates'.

• the WDCM_Sqoop_Clients.R procedure fails for some databases;
• however, this seems to be happening on occassion only;
• inspect and solve.

@Christine_Domgoergen_WMDE Your second follow-up report (May 20 - May 26):

Ok, here's what I've got:

Following thorough experiments with Python Gensim and Apache Spark, which both use the same online LDA estimation, these are the conclusions:

@Christine_Domgoergen_WMDE Your first follow-up report (May 13 - May 19):

@JAllemandou Thanks for feedback!

@Lea_WMDE Here we go:

@Lea_WMDE Yes, we do have a more or less steady increase in mobile edits on Wikidata:

@Lea_WMDE Here's what was happening with the mobile edits since the beginning of the year. Note: the last data point is May 2019, it's incomplete of course.

@JAllemandou You're the man, thank you, I see now that revision_tags is a new field since the 2019-04 (April 2019) snapshot of mediawiki_history:

WD Percent Usage Dashboard: done.

@Lea_WMDE from Analytics/Data Lake/Edits Wikitech documentation page:

• fixed.

• Fix client-side dependencies in the Percentage of articles making use of data from Wikidata dashboard.

• WDCM general documentation update on Wikitech - Completed.

• essential properties/classes for languages - for the Wikidata Languages Landscape #T221965 --> this will be transferred as a sub-task to #T221965.

• essential properties/classes for taxa (WDCM) - DONE.

• essential properties/classes for organizations (WDCM) - DONE.
• essential properties/classes for geographical objects (WDCM) - DONE.

Daily report(s) for re:publica 2019 conference: