@Lydia_Pintscher @RazShuty @WMDE-leszek Here's a glimpse of what we've found out thus far:

@Stefan_Schneider_WMDE I think we should meet:

@Stefan_Schneider_WMDE I think we should meet:

@Theklan As of the following

@Theklan Hello, I am the maintainer of the WDCM system. Ok, let's inspect what is happening here:

• working on analytics/visualizations now;
• next steps: dashboard.

• Following the latest WDCM update ruwiki is back;
• still cannot localize what exactly happened to the previous update.

@Aklapper Thanks for noticing this.

Check out what is happening with:

@Halfak Thank you, Aaron.

Amir, right before our meeting, what we need here is simple:

Something to begin with:

• the Jaccard similarity and distance matrices: testing, the procedure is memory efficient but slow (subsetting the dgCMatrix class matrix...):
• DONE. We can have the Jaccard distances here too.

• Batch processing over sparse matrices (dgCMatrix class) is now employed to compute
  • the co-occurence data set: success, using approx. order of magnitude less resources than the previously employed procedure, and
  • the Jaccard similarity and distance matrices: testing, the procedure is memory efficient but slow (subsetting the dgCMatrix class matrix...).

@Lea_WMDE Hm, this might be the solution - dygraph se to dylegend(show = 'follow'), please check: http://wmdeanalytics.wmflabs.org/WD_pageviewsPerNamespace/
Note. This was the initial solution and there is one thing I don't like about it in spite of the fact that it solves the problem that you were facing (focus overlapping title).
Let me know if you like this approach better, please.

Strange.
Lea, please let me know what browser are you using. I have tested the dashboard on Chromium and Mozilla Firefox under Ubuntu; the WDCM system, using the same front-end technology (RStudio Shiny) was tested over an even broader range of browsers (including macOS), and at this point I cannot really tell what is causing the problem - but I will do my best to figure it out.

• Given how often is stat1007 used by us analysts,
• it barely has the resources for the computations that we need here (the languages x languages contingency table; takes at least ~25Gb to compute);
• Apache Spark cannot help (large number of categories -> Spark's dataframe functions cannot support contingencies in such situations).

Could you add the info why wikistats2 data differs from these graphs to the explanatory text?

Done: dashboard.

@Lea_WMDE The Total Average is back: dashboard.

@Milimetric Thanks for the clarification, Dan.
@Lea_WMDE This implies that

@Lea_WMDE Ok, here is a direct test (Pyspark code against the wmf.pageviews_hourly table):

@Lea_WMDE So that is one order of magnitude and looks straightforward impossible to happen. Please let me check.
I guess the difference of this magnitude could not be a consequence of the fact that we have picked only four namespaces (Entity, Property, Lexeme, and EntitySchema)? Please confirm.

@Lea_WMDE Take a look, please: http://wmdeanalytics.wmflabs.org/WD_pageviewsPerNamespace/

@Lea_WMDE I am on it.

• On the vertical axes the dashboards now uses K, M, and B for thousands, millions, and billions of pageviews, respectively.

• The dashboard is now running a regular daily update;
• fixing the axis labels now.

@Lea_WMDE I am on it, putting the dashboard on regular updates + fixing the labels to include decimal points.

As stated in the README.txt:

@Lydia_Pintscher

That's "our" information. And then we have links/external identifiers to say 3 libraries that also have information about X. We want to somehow quantify the latter for all of Wikidata's entities.

Q. Do I understand correctly: you would like to have some sort of comparison (a "ratio" of some form) between (a) knowledge on X in Wikidata and (b) knowledge on X in other (linked from Wikidata) databases?

We should try to find ways to quantify this information.

@Lea_WMDE You can now test your new dashboard: http://wmdeanalytics.wmflabs.org/WD_pageviewsPerNamespace/

• data set review requested from Analytics in T227905;
• next steps:
  • visualizations + dashboard;
  • test, deploy.

• data set production - completed.

@Lea_WMDE I guess 640 is the EntitySchema namespace (figured this out from this Gerrit patch, since it is not documented in the Wikidata namespaces), right?

• Deployed everything.

@Christine_Domgoergen_WMDE Then please close this ticket and open a new one when the campaign is ready. It will take me no time to set up the analytics for the new campaign if only parameters like landing pages are different. Thank you.

@Christine_Domgoergen_WMDE Will we be using exactly the same parameters as in this campaign, or are we talking about a new, different campaign altogether?

2019/06/21: no pageviews, no user registrations.
2019/06/22: no pageviews, no user registrations.
2019/06/23: no pageviews, no user registrations.
2019/06/24: no pageviews, no user registrations.
2019/06/25: no pageviews, no user registrations.
2019/06/26: no pageviews, no user registrations.
2019/06/27: no pageviews, no user registrations.

@bd808

2019/06/14: pageviews: staa = 2, strz = 1 , user registrations: 0.
2019/06/15: pageviews: strz = 1 , user registrations: 0.
2019/06/16: pageviews: none, user registrations: 0.
2019/06/17: pageviews: none, user registrations: 0.
2019/06/18: pageviews: none, user registrations: 0.
2019/06/19: pageviews: strz = 1, user registrations: 0.
2019/06/19: pageviews: strz = 1, user registrations: 0.

@Lea_WMDE Do we have any additional requirements here or shall we resolve the ticket?

• The only two dashboards that now depend upon WDCM_updateLabs.R are

• Refactored.

• wdcm_maintable removed from hdfs;
• all WDCM dashboards now running Apache Spark supported update engines;
• resolved.

The WDCM Biases dashboard is now back on updates:

• it will be update once monthly, and
• it's update is dependent upon the most recent version of the WD dump copy in hdfs (see T209655);
• the dashboard is now fully client-side dependent,
• and because it now uses Spark as its ETL back-end we can finally remove
• the huge wdcm_maintable from HDFS that previously supported our ETL procedures for the WDCM system (Hive; see T214586).

• Goodbye Advanced Search Extension dashboard :(

• archive the Advanced Search Extension dashboard (tracking).

@JAllemandou Thanks for the recent 20190603 dump copy in HDFS.

• Pyspark ETL procedures for WDCM Biases completed;
• next step: re-factor R code to work with Spark outputs.

• The WDCM Biases Dashboard's back-end has to switch to Apache Spark;
• it maintains large data sets, depends upon WDQS heavily by sending queries that time-out every now and then; beyond that,
• it relies on geo-coordinates which, when fetched together with items from WDQS, produce long vectors that cannot be converted from raw to character() in R (and which then need to be parsed by {jsonlite} into readable data.frame types for processing).

• Next step: completely remove WDCM_updateLabs.R from the WDCM pipeline.

• WDCM_Process.R is now deprecated.

• The dashboard is now fully client-side dependent;
• update_labs module - hopefully to be fully deprecated soon - updated.

@elukey I was able to reduce the amount of disk usage in my home directory on stat1007 by approx. 40Gb.
My apologies: there were some old, raw data sets there that I have forgot to clean up...

@elukey I'm on it.

Of course. Please state exactly (FROM - TO) the dates for the remaining weekly reports. Thanks.

• The WDCM Biases Dashboard is the only remaining dashboard whose back-end relies on this Hive table.
• As soon as the changes are implemented there, the table will be removed from hdfs.

@Christine_Domgoergen_WMDE Can we close this ticket now?

• All changes implemented: {text2vec} procedures replacing {maptpx};
• Deploying now; the test run was successful so the dashboards are already update from the new ML back-end;
• first run in production scheduled for tomorrow, June 7/2019;
• behavior: too many topics in the optimal model from perplexity based model-selection;
• action: we will switch the WDCM model selection procedures from perplexity based to coherence based (as in WDCM Sitelinks and Titles).

• WDCM Geo is back on updates, re-designed, fully client-side dependent, and does not rely on the wdcm_maintable anymore (see T214586):

http://wmdeanalytics.wmflabs.org/WDCM_GeoDashboard/

• WDCM Geo Dashboard does not depend upon this table anymore.

• WD_percentUsageDashboard update time changed: from 08:00 UTC to 12:00 UTC.

@Lydia_Pintscher I got it:

• Correction. It is not the Apache Sqoop procedure in WDCM_Sqoop_Clients.R that fails (luckily!);
• explanation: I've stumbled into an unfinished log file when I was checking this... and some databases, naturally, where missing - because the procedure did not complete all the passes yet.
• Continue with: inspect why the usage number reported on WD_percentUsageDashboard 'oscilates'.

• the WDCM_Sqoop_Clients.R procedure fails for some databases;
• however, this seems to be happening on occassion only;
• inspect and solve.