@Lea_Lacroix_WMDE Thank you!

@Merle_von_Wittich_WMDE The tracking procedures for this are in place and running smoothly. Do we need this ticket anymore?

• And in production: https://wikidata-analytics.wmcloud.org/app/WDCM_StatementsDashboard
• Not yet included from the Wikidata Analytics Portal.

• Deployed (pre-production) on test-server: http://datakolektiv.org/app/WDCM_StatementsDashboard
• To be deployed in production on https://wikidata-analytics.wmcloud.org/ tomorrow

Do we need this ticket still opened and the old system's VM running?

@elukey All checked - all superfine. Thank you very much again for your intervention!

@elukey Do you mind if we wait for the next regular update of our Wikidata Analytics and see then? It takes place in less than a week.

@JAllemandou Anytime.

@WMDE-leszek @Lydia_Pintscher The updates are unblocked now; we expect everything to be found back in the expected state until February 21st early hours.

we can keep stat1004 in this state for the weekend so your regular update goes through, we'll revert it only when a final solution will be found. Would it be ok?

Perfect. I suggest that we keep this ticket opened with a lowered priority until you decide upon the final state of stat1004 and other clients that might use Sqoop?

@elukey It works! Thank you! Q:

@elukey Thank you very much for a prompt reaction! - testing now and getting back to you.

@elukey I think it is the Apache Sqoop call that fails. Example:

@elukey Thank you. I think now that we were facing a similar problem already - your comment in T274866#6840079 has just reminded me of it... Please give me some time to me search the Phabricator a bit and try to remember what has happened then and how it was solved. In effect, what happens is that WDCM_Sqoop_Clients.R - which just orchestrates Apache Sqoop and HiveQL calls from stat1004 - ends up producing an empty goransm.wdcm_clients_wb_entity_usage table for some reason.

@elukey Unfortunately...

@elukey First of all: thanks for reaching out!

@Merle_von_Wittich_WMDE No worries, I will follow the developments and take over once the campaign is ready for test and tracking.

Everything in relation to T261905#6688736 is completed.
I guess we could inform the community about the new Wikidata Analytics and begin the transition.

• If you visit the public data directory at https://analytics.wikimedia.org/published/datasets/wmde-analytics-engineering/Wikidata/WD_Inequality/
• you will find one .csv file there:

• the computation of the Hoover inequality index will be run every time a new snapshot of wmf.mediawiki_history is detected,
• in Pyspark for ETL, orhestrated by a Python script that checks the snapshot, runs the ETL, and computes the index;
• the data will be served as a .csv file from the public directory,
• and the future dashboard will be client-side dependent and use the public dataset to visualize the results.

@Lydia_Pintscher @Jan_Dittrich On the test server, of course - no WMF or WMDE domains are there.

I do not know how costly running "begin of time" is – I am pretty fine with running it every 3 month or so, too to save some computing time and CO2.

The planet would survive a monthly update. Many of our system run similar, regular monthly updates, so this can be packed on the same train.

Everything is ready to put this on regular updates. There are some constraints in relation to the cost of data engineering procedures. Let me share with you:

@WMDE-leszek Here is what I am going to do for starters:

@WMDE-leszek This is not going to work the way it is in production now. We need a way to be able to update real-time, and still stay in the same production environment as the rest of the Wikidata Analytics. As I have predicted, productionizing this (rather simple software) is going cause us a problem.

@WMDE-leszek The problems in relation to URL rendering in production is solved. A minor bug was in question - as I hoped it will be.

This system is running in production from: https://wikidata-analytics.wmcloud.org/app/QURATOR_CurrentEvents

The Qurator Curious Facts is now running in production: https://wikidata-analytics.wmcloud.org/app/QURATOR_CuriousFacts

• All URLs in the Wikidata Analytics WMF documentation pages were changed as per @Lea_Lacroix_WMDE's request.
• The new, Wikidata Analytics repo is ready.
• There will be some minor changes here in the near future, but we are definitely ready to go with the announcement of the new Wikidata Analytics.

The dashboard is running on the test server now: http://datakolektiv.org:3838/WDCM_Statements/

Public datasets:

The new Wikidata Analytics runs in production from our CloudVPS: https://wikidata-analytics.wmcloud.org/

Everything in relation to T261905#6688736 is completed.
I guess we could inform the community about the new Wikidata Analytics and begin the transition.

The campaign update is available, but still no changes in user registrations and edits (no campaign edits).

2021/01/09 - WDCM ETL/ML Engines (back-end, stat100* machines) productionized w. {renv}.

The update until 2021/01/05 is in the campaign public data directory: one user registration only, and still no edits.

The intro says the last update of user edit data is from November 24th, is that true?

No, it is not: the observation window for this campaign ends on December 24th as requested, and the datasets are aligned with that.
Please find the report with the corrected Intro here:

@Christine_Domgoergen_WMDE Exactly, sorry about that. I will updated it very soon and share the final report here with you.

NOTE. 2021/01/02 and still no user registrations for this campaign. The pageviews dataset is update and available from the public data directory.

The dashboard is running on the test-server: http://datakolektiv.org:3838/WD_CuriousFacts/

So one batch up to 31.12. and then one from 01.01.?

@Christine_Domgoergen_WMDE Done.

@Merle_von_Wittich_WMDE Finally, user edits tested, all fine, found:

@Merle_von_Wittich_WMDE User registrations tested, found:

• Pageviews tested, working;
• Testing user registrations, edits now.

The requests presented in T202610#5535326 and T202610#5535427 are unclear.
Please define the requests in a clear and concise language (I am discouraging the use of mathematical or technical language, but it must be made understandable and refer directly to the data products offered at the Wiktionary Cognate dashboard), and then we can consider them.

Working on this is not feasible at this point.
The idea to have a multilingual UI implemented in the WDCM system will have to wait for another cycle of system redesign planning.

@Merle_von_Wittich_WMDE Thank you very much. Accepted :)

@Merle_von_Wittich_WMDE Today 16:00 CET, how about it?

Nice to meet you, Merle.

@Jan_Dittrich Got it. I will get back to you if it turns that I need more info.

@Lydia_Pintscher Can we resolve this ticket or do we need anything else here?

The URLs issue - where the URL did not change from the landing page of the new Wikidata Analytics services - is resolved.
@Lea_Lacroix_WMDE This means that our users will be able to bookmark any dashboard directly.

@Lydia_Pintscher Here it goes:

@Lydia_Pintscher Of course, it will be produced and posted here during the day.