[EPIC] Migrate selected R-based Wikidata products
Open, LowPublic
Actions

Assigned To

Authored By

	• Manuel
	Mar 23 2023, 2:56 PM

Description

What is this task?
This task is used for planning and organizing only.

Description of the main objective:

see PRD

Milestones:
Phase 1: Evaluate

Understand the current state
Decision about Spark3 migration (high priority)
Document the dashboards
Evaluate the legacy data products (low priority)
Decisions about legacy data products (low priority)

For details see PRD

Tools for Wikidata editors
https://wikidata-analytics.wmcloud.org/app/CuriousFacts
https://wikidata-analytics.wmcloud.org/app/CurrentEvents

Tools for Wictionary editors
https://wiktionary-analytics.wmcloud.org/Wiktionary_CognateDashboard (see for details T321666)

Documentation of the codes can be found in the doc linked here.

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		Lydia_Pintscher	T332899 [EPIC] Migrate selected R-based Wikidata products
Open	BUG REPORT	None	T327922 All of our ShinyProxy-based dashboards have stopped working
Open	BUG REPORT	None	T321666 Wiktionary Cognate Dashboard is not accessible
Resolved		AndrewTavis_WMDE	T340718 Investigate prior WMDE analytics tables / assets
Resolved		AndrewTavis_WMDE	T334951 Wikidata Concepts Monitor ETL Migration to Spark3
Duplicate		None	T342531 Use-cases for our R-based analytics products
Open		None	T353175 Show a message instead of broken Shiny dashboards
Resolved		AndrewTavis_WMDE	T358254 [Analytics] Investigate effort of selective legacy migrations to Airflow
Open		AndrewTavis_WMDE	T360296 [Analytics] Implement data process to identify missing Wiktionary entries

Event Timeline

• Manuel created this task.Mar 23 2023, 2:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 23 2023, 2:56 PM

• Manuel renamed this task from [EPIC] Save WDCM to [EPIC] Save our legacy products (WDCM and Cognate dashboard) .Mar 23 2023, 2:57 PM

• Manuel added a subtask: T332898: Wikidata Concepts Monitor ETL migration to Spark3.

• Manuel moved this task from Incoming to Waiting Epics on the Wikidata Analytics board.

• Manuel added a subtask: T321666: Wiktionary Cognate Dashboard is not accessible .Mar 23 2023, 3:10 PM

• Manuel added a subtask: T251792: Wiktionary Cognate Dashboard not updated.

• Manuel added a subtask: T327922: All of our ShinyProxy-based dashboards have stopped working .

• Manuel moved this task from Waiting Epics to Backlog on the Wikidata Analytics board.Mar 23 2023, 3:17 PM

Maintenance_bot added a project: Wikidata.Mar 23 2023, 3:30 PM

Pamputt subscribed.Mar 23 2023, 6:06 PM

• Manuel removed a subtask: T321666: Wiktionary Cognate Dashboard is not accessible .Apr 1 2023, 9:17 AM

• Manuel renamed this task from [EPIC] Save our legacy products (WDCM and Cognate dashboard) to [EPIC] Save our R-based Wikidata products .Apr 1 2023, 9:26 AM

• Manuel updated the task description. (Show Details)

• Manuel removed a subtask: T251792: Wiktionary Cognate Dashboard not updated.Apr 1 2023, 9:30 AM

• Manuel moved this task from Backlog to Waiting Epics on the Wikidata Analytics board.Apr 1 2023, 9:41 AM

• Manuel claimed this task.Apr 1 2023, 9:52 AM

• Manuel updated the task description. (Show Details)Apr 11 2023, 9:46 AM

AndrewTavis_WMDE updated the task description. (Show Details)Apr 20 2023, 2:48 PM

AndrewTavis_WMDE updated the task description. (Show Details)

Added a link to the task description pointing people to the following doc where I've been documenting the code base: https://docs.google.com/document/d/1aYWcStCZySbKTKbHdFzb4n9ZBpBntIbSZsnd99nriuc/edit?usp=sharing

JeanFred subscribed.Jun 20 2023, 7:36 PM

I was pointed to this ticket by @Lydia_Pintscher: I wanted to update a presentation slide that uses https://wikidata-analytics.wmcloud.org/app_direct/WD_ExternalIdentifiersDashboard (that’s the only dashboard I can remember using)

Hi @JeanFred 👋 Thanks for reaching out :)

I'm currently in the midst of trying to get external identifiers running locally at the moment, but I'm running into some problems that didn't occur with the rest of WDCM. Is there a date that you need the information by? I'm planning on sitting with a coworker tomorrow to look at it a bit more in depth.

Hi @JeanFred, the data source for the dashboards is still updating (for now). So you may be able to find the information that you are looking for here.

I am currently investigating which parts of the dashboards were actively used. So I would be curious about what data exactly was helpful for you on that dashboard.

Ah yes, good thought on that, @Manuel! Just didn't make the connection :) Thanks! :D

• Manuel renamed this task from [EPIC] Save our R-based Wikidata products to [EPIC] Migrate our selected R-based Wikidata products .Jul 4 2023, 1:56 PM

• Manuel updated the task description. (Show Details)

• Manuel added a subtask: T340718: Investigate prior WMDE analytics tables / assets.

• Manuel added a subtask: T334951: Wikidata Concepts Monitor ETL Migration to Spark3.Jul 4 2023, 1:59 PM

• Manuel removed a subtask: T332898: Wikidata Concepts Monitor ETL migration to Spark3.

• Manuel renamed this task from [EPIC] Migrate our selected R-based Wikidata products to [EPIC] Migrate selected R-based Wikidata products .Jul 4 2023, 2:07 PM

• Manuel moved this task from Waiting Epics to Kanban on the Wikidata Analytics board.

• Manuel edited projects, added Wikidata Analytics (Kanban); removed Wikidata Analytics.

• Manuel moved this task from Incoming to WIP Epics on the Wikidata Analytics (Kanban) board.

• Manuel updated the task description. (Show Details)Jul 4 2023, 2:10 PM

• Manuel added a project: Epic.

Hey @AndrewTavis_WMDE & @Manuel , I only used the dashboard to get a nice visualization of the external ID galaxy − see this slide (the red bubble is the video-game related IDs)

Thank you for your feedback! :)

• Manuel updated the task description. (Show Details)Jul 5 2023, 9:21 AM

• Manuel closed subtask T334951: Wikidata Concepts Monitor ETL Migration to Spark3 as Resolved.Jul 7 2023, 9:14 PM

• Manuel renamed this task from [EPIC] Migrate selected R-based Wikidata products to [EPIC] Migrate selected R-based Wikidata products [up to milestone 3].Jul 7 2023, 9:19 PM

MarcoSwart subscribed.Jul 8 2023, 3:43 PM

• Manuel edited projects, added Wikidata Analytics; removed Wikidata Analytics (Kanban).Jul 20 2023, 10:13 AM

• Manuel closed subtask T340718: Investigate prior WMDE analytics tables / assets as Resolved.Jul 24 2023, 8:27 AM

• Manuel updated the task description. (Show Details)Jul 24 2023, 11:13 AM

• Manuel mentioned this in T342531: Use-cases for our R-based analytics products .Jul 24 2023, 11:27 AM

• Manuel added a subtask: T342531: Use-cases for our R-based analytics products .

• Manuel moved this task from Incoming to Waiting Epics on the Wikidata Analytics board.Jul 26 2023, 7:55 AM

• Manuel renamed this task from [EPIC] Migrate selected R-based Wikidata products [up to milestone 3] to [EPIC] Migrate selected R-based Wikidata products .Feb 7 2024, 11:45 PM

• Manuel updated the task description. (Show Details)

• Manuel added a subtask: T358254: [Analytics] Investigate effort of selective legacy migrations to Airflow .Feb 22 2024, 5:22 PM

• Manuel edited projects, added Wikidata Analytics (Kanban); removed Wikidata Analytics.Feb 28 2024, 1:57 PM

• Manuel closed subtask T358254: [Analytics] Investigate effort of selective legacy migrations to Airflow as Resolved.Mar 18 2024, 11:14 AM

AndrewTavis_WMDE changed the status of subtask T360296: [Analytics] Implement data process to identify missing Wiktionary entries from Open to Stalled.Jun 6 2024, 11:13 AM

• Manuel triaged this task as Low priority.Jul 24 2024, 9:13 AM

• Manuel reassigned this task from • Manuel to Lydia_Pintscher.Jul 30 2024, 8:57 AM

karapayneWMDE moved this task from Kanban to Epics on the Wikidata Analytics board.Aug 21 2024, 3:20 PM

karapayneWMDE edited projects, added Wikidata Analytics; removed Wikidata Analytics (Kanban).

AndrewTavis_WMDE changed the status of subtask T360296: [Analytics] Implement data process to identify missing Wiktionary entries from Stalled to Open.Sep 5 2024, 12:43 PM