Ok, here's what I've got:

Following thorough experiments with Python Gensim and Apache Spark, which both use the same online LDA estimation, these are the conclusions:

@Christine_Domgoergen_WMDE Your first follow-up report (May 13 - May 19):

@JAllemandou Thanks for feedback!

@Lea_WMDE Here we go:

@Lea_WMDE Yes, we do have a more or less steady increase in mobile edits on Wikidata:

@Lea_WMDE Here's what was happening with the mobile edits since the beginning of the year. Note: the last data point is May 2019, it's incomplete of course.

@JAllemandou You're the man, I see now that revision_tags is a new field since the 2019-04 (April 2019) snapshot of mediawiki_history:

WD Percent Usage Dashboard: done.

@Lea_WMDE from Analytics/Data Lake/Edits Wikitech documentation page:

• fixed.

• Fix client-side dependencies in the Percentage of articles making use of data from Wikidata dashboard.

• WDCM general documentation update on Wikitech - Completed.

• essential properties/classes for languages - for the Wikidata Languages Landscape #T221965 --> this will be transferred as a sub-task to #T221965.

• essential properties/classes for taxa (WDCM) - DONE.

• essential properties/classes for organizations (WDCM) - DONE.
• essential properties/classes for geographical objects (WDCM) - DONE.

Daily report(s) for re:publica 2019 conference:

Daily report(s) for re:publica 2019 conference:

@Milimetric Would you happen to know if there is a convenient method to differentiate between (a) edits made from mobile vs. (b) edits made from desktop, in a particular project (say, Wikidata)? Thank you.

Another possibility would be to parse the X-Analytics field of the wmf.webrequest table and look into the values of the mf-m key:

Unfortunately, our edits data currently do not encompass any fields that would allow us to separate edits made from mobile vs. desktop.

Daily report for re:publica 2019 conference, May 7, 2019:

If the growth is natural, would it be a valid assumption to assume that the editing behavior for mobile also increased?

Daily report for re:publica 2019 conference, May 6, 2019:

... to create your own Python virtual environment on stat1007 and pip install the package ...

@Ottomata Hey, Otto

• These operations are meant to replace all R orchestrated, massive, and time consuming Wikidata API/WDQS SPARQL calls from WDCM and related dashboard back-ends;
• The following datasets were produced until now:
  • WD labels for the top 15 languages per number of speakers (essential for the WDCM system);
  • Q5 (Human): all items + the essential properties for our WD statistical systems:
    • P21
    • P106
    • P170
    • P50
    • P101
    • P27
    • P39
    • P103
    • P1412
    • P172
    • P463
    • P1344

@Christine_Domgoergen_WMDE Test complete, all fine:

@Christine_Domgoergen_WMDE Testing now.

I have inspected all mobile pageviews of Wikidata for April 2019.

Can we make a registration test just before our meeting? I would make a test registration and you could then tell me, if you received the data. Is that okay for you!

It's definitely not Googlebot (Smartphone), I've checked the wmf.webrequest for a sample:

0. First of all, I started by replicating the Wikistats data (https://stats.wikimedia.org/v2/#/wikidata.org/reading/total-page-views/normal|bar|2-Year|access~mobile-web) that you have shared in the ticket description, to ensure that we are looking at the same data. I have used the Projectview hourly Hive table and the replication is perfect.

• Fundamental dataset (language code x language code similarity/distance matrix) produced from WD JSON dump, hdfs copy w. Pyspark;
• PoC completed; can be done;
• details forthcoming.

@Christine_Domgoergen_WMDE Well it makes sense to be fast on this since it's holiday in Germany tomorrow and our test should take place the day after, right?

• I think that I understand everything, and
• I have only two questions:

@Christine_Domgoergen_WMDE Of course we can test before our meeting!

@Lydia_Pintscher \o/ Oh, yes, please do :)

• all computations that could have been moved to the back-end are now there;
• the dashboard is not fully client-side dependent anymore (it wasn't realistic in the first place, mea culpa; too large datasets).

• The aesthetics are back:
  • {igraph} MDS layout deprecated in favor of
  • {igraph} Fruchterman-Reingold algorithm;
  • this solution is slower and I will try to optimize it as much as I can, but
  • the result is stunning.

@Lydia_Pintscher

• Everything else takes place once the WD JSON dump copy to HDFS (T209655) is in production, and the Analytics-Engineering tell me that is going to take a while.
• I think we should consider investing a bit more of my time here to optimize the dashboard (large datasets --> heavy on client-side processing). Please let me know what you think.

• Implementing changes in the WD external identifier class visualizations: DONE;
• in relation to T204440#5097057, a compromise was introduced:
  • the WD identifier class network is generated to encompass all identifiers who belong to the class either by P31 or by P279 paths;
  • the table to the right of the network visualization will list only identifiers that belong to the class in a P31 sense.

• implement the suggestion by @Pintoch (see: T204440#5097057)
  • data structure: DONE
  • implementing changes in the WD external identifier class visualizations now;
• Next:
  • check why 20 - 30% of data are not delivered from Spark; most probable cause: failures due to I/O operations;

• revive the Overlap Network visualization - DONE

• there are some minor interventions that need take place in the visualizations code. - DONE
• Next steps:
  • revive the Overlap Network visualization;
  • check why 20 - 30% of data are not delivered from Spark; most probable cause: failures due to I/O operations.

• Back-end re-factored; dashboard online, not all functionality complete:
  • the Overlap Network tab will have to wait until I figure out why we don't get all of the data from Spark;
  • there are some minor interventions that need take place in the visualizations code.

@Envlh It is not fixed yet. I am getting more data, but not all of it, see: T204440#5116460

Overview/Status Report:

@agray The SPARQL queries indeed select distinct items per property x property intersection... back to the drawing board: what is missing in my data?
I will have to dig deep to find out, the Pyspark ETL code for this dashboard is already looking into all statements, claims, and references. I fear it might be related to non-deterministic operations in Spark that could have affected the completeness of the datasets, but I've done everything to prevent such effects. I'm a bit puzzled, but I will find out the cause this or the other way. Thank you very much for testing.

@Jheald Given that we discard all cases of multiple use of the same identifier with a particular item, does the number seem reasonable to you?

Overview/Status report for this task:

• Data engineering test: success;
• tSNE running now.

• Data engineering procedures code re-factored and in place;
• testing now.

@agray Got it. Thank you. Still working on the overlap dataset.

I can confirm that the numbers on the tables seem a bit off for some other properties. I've been looking at P1614 (History of Parliament), which is complete and fairly stable. It currently has 21428 IDs on 17942 items (there's a lot of items with two/three IDs) and hasn't had any big changes since I finished matching in mid-2018.