@Stefan_Schneider_WMDE :) That was not really fast, Stefan, thank you for you kind words but @Christine_Domgoergen_WMDE posed her questions day ago. But I really had to prioritize some WD related things.

I have a question about the table 3.1 User edits: daily: the days 2019-01-19, 2019-01-20, 2019-01-27 and 2019-01-29 seem to be missing, could you have a look? Thank you!

I am getting back to this analysis during the day.

• The central component of the WDCM CloudVPS update is now fully re-factored + xml configuration file is ready;
• this change affects:
  • WDCM Overview Dashboard
  • WDCM Semantics Dashboard
  • WDCM Usage Dashboard;
• next steps: WDCM_Geo, WDCM_Biases, WDCM (S)itelinks, and WDCM (T)itles.

• TW Advanced Search Extension update engine is now re-factored from the WDCM CloudVPS update engine;
• Next step: xml config/clean-up for the WDCM CloudVPS update engine.

• Wiktionary Cognate Dashboard now has an independent update engine in CloudVPS;
• testing now.
• next step: TW Advanced Search Extension update engine: re-factor from WDCM.

• WDCM Semantics dashboard is partially refactored.

• Next steps: re-design and re-factor (T214337):

• WDCM Semantics is now re-factored with an xml config file

@bd808 They are unused and can be safely deleted.

@Addshore I do understand what is happening but I am still unsure of the effects. Thanks for the docs. Finally, and while you're already here:

@Ladsgroup Thanks mate. Another question (for you or @Addshore):

• Fixed (minor issue);
• everything should be fine in few hours or so.

@Addshore @Ladsgroup Do you have any idea on when will the proposed changes in the DB layout take effect?

• Re-designed WDCM Semantics dashboard deployed;
• N.B. See: T180891#4925063 (critical);
• Next step: re-factor CloudVPS update(s).

• WDCM Semantics Dashboard: {rbokeh} issue detected:

Solved.

@JAllemandou @Lydia_Pintscher @Addshore I can access the JSON WD dumps from hdfs now.
I should be able to solve this in Pyspark. Reporting back as soon as I have something.

@Addshore wdcm.wmflabs.org is now removed from wikidataconcepts and re-created at wmde-dashboards where it maps 3838 (Shiny server default).

@Addshore Got it. I will ping back as soon as I remove the domain from wikidataconcepts.

@Addshore Noot on wikidataconcetps, but wmde-dashboards.

@joal Let me take a look at it and I'll get back to you. Thanks!

Oh... Ok I am on it.

@Lydia_Pintscher Depends :) My proposal in T214897#4918562 does not make things easier for me in any way, on the contrary.
But I guess we should give it a try if other approaches (e.g. someone parsing the dump w. WDTK to produce the dataset, having the WD dump in the Data Lake) are more expensive.
Let me know what you think.

@Addshore Exactly (@Lydia_Pintscher - We have already discussed similar approximate solutions).

@Christine_Domgoergen_WMDE The updated campaign report is ready:

• Prioritizing this task again;
• all requests and issues reported in T211690#4918152 will be taken into consideration in the updated campaign report (edits until Jan 30);
• reporting back as soon as I have the updates.

• WDCM Usage and Overview dashboards are (partially) refactored now;
• still running a single update engine for all WMDE CloudVPS engines.

• New WDCM Overview dashboard deployed;
• Next steps: deploy new WDCM Semantics dashboard.

Hey: don't you think that we could get this from the pagelinks table in Wikidatawiki's database?

When I sum up all banner impressions from 1.1.1 Banner Impressions Overview: Table the result is 168.540.386, when I sum up all impressions in 1.1.2 Total Banner Impressions (split by device) the result is 168.540.422. Could you check, why the numbers differ here?

The correct number is: 168,540,386 - the first one that you have reported. Both tables have the same total number of impressions reported (checked by re-running the report code). Did you add up the number correctly?

@Christine_Domgoergen_WMDE I am on it.

• user rstudio-untrusted can now run all WDCM dashboard on CloudVPS for testing purposes;
• in "production" they are deployed from a goransm controlled crontab on CloudVPS; this is crazy
• but it is a necessary measure in respect to T213977.

Hi @Framawiki, and thanks for a prompt response.

@abian The code is in place. Just ping me somewhere whenever you need an update.

@abian You're welcome! Your analysis is awesome and could be used to exemplify how a Client/Manager/Editor/Owner should introduce the problem to a Data Scientist/Analyst!

@abian @Lydia_Pintscher We have the results.

• we have the dataset (WD items, and for each item the number of pages which make use of it),
• and we're running the power-law estimation procedures now; as predicted
• that is going to take a while.

@Christine_Domgoergen_WMDE Here is the report with user edits and training modules data included.

Ruby? No, thanks - R, Python, SQL, Spark whatever, but not another programing language on my agenda.

Currently I have no idea on (a) where do the data live, and (b) what exactly is done do get the relevant dataset, but if @Ragesoss can provide a concise tutorial (a Google Hangouts session, say) - I am more than willing to learn.

@Christine_Domgoergen_WMDE Here is the preliminary campaign report. Findings on user edits and training modules will be included on January 24th. The final report will be delivered immediately after January 30th to encompass all user edits two weeks following the end of the campaign.

I have the dataset with me, so I guess we can close this ticket now.

• Following the necessary changes (see T204695), this will continue on the new wmdeanalytics.eqiad.wmflabs project.
• Next step: re-design the Shiny portal/entry page to encompass all WMDE analytics.

• Wikidata % of articles usage Dashboard: http://wmdeanalytics.wmflabs.org/WD_percentUsageDashboard/

• WDCM (T)itles: http://wmdeanalytics.wmflabs.org/WDCM_TitlesDashboard/
• Wiktionary Cognate: http://wmdeanalytics.wmflabs.org/Wiktionary_CognateDashboard/

• WDCM (S)itelinks: http://wmdeanalytics.wmflabs.org/WDCM_SitelinksDashboard/

@Pamputt @Lea_Lacroix_WMDE Fixed.
NOTE The dashboard's URL is changed, please update: http://wmdeanalytics.wmflabs.org/Wiktionary_CognateDashboard/

• TW_AdvancedSearchExtension has a new URL: http://wmdeanalytics.wmflabs.org/TW_AdvancedSearchExtension/

• WDCM Biases: http://wmdeanalytics.wmflabs.org/WDCM_BiasesDashboard/

• WDCM Geo Dashboard's new URL will be: http://wmdeanalytics.wmflabs.org/WDCM_GeoDashboard/;
• it's running there.

• WDCM Usage is now found on http://wmdeanalytics.wmflabs.org/WDCM_UsageDashboard/

• WDCM Semantics is running from: http://wmdeanalytics.wmflabs.org/WDCM_SemanticsDashboard/

• WDCM Overview Dashboard is now running from: http://wmdeanalytics.wmflabs.org/WDCM_OverviewDashboard/
• NOTE: the dashboard is still running from the wikidataconcepts instance too.

• I think I found the solution how to keep the R development workflow from the new wmde-dashboards.eqiad.wmflabs instance;
• Please ping here *before* you delete the wikidataconcepts.eqiad.wmflabs instance. Thank you.

@Christine_Domgoergen_WMDE Of course. Let's keep this ticket open and encompass the final report beyond the daily ones here.