Maniphest T198776

Set up data infrastructure for program metrics
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	nshahquinn-wmf
	Jul 4 2018, 12:54 AM

Description

We've started to plan how we'll calculate the Audiences 2018–19 annual plan metrics, and we've identified some infrastructural needs.

Daily editing data, with tags, in the Data Lake

Filed

Need some type of aggregated and/or denormalized table of edits that contains data about edit tags (to be used for mobile retention, mobile edits and editor counts, or anything else that relies on tagging edits)
- Needs to be granular enough to look at short-term editor retention (daily is probably sufficient)
We need it updated relatively quickly, within a day after the events take place (if the Data Lake is only source and it isn't available till 10th after month's end, that is not tenable).
A one-time sqoop of the change_tag tables into the Data Lake to provide past edit tag data to complement current data

Identifying reverts

In mediawiki_history, we'd calculate reverts using the convenient is_reverted field. How do we do it when we simply have a table of revisions with their hashes?

Related Objects
Search...

Status	Assigned	Task
Declined	nshahquinn-wmf	T198776 Set up data infrastructure for program metrics
Declined	nshahquinn-wmf	T198777 Pilot new editor retention and mobile new editor retention calculations
Resolved	• Pchelolo	T201062 Load change tags into the Analytics Data Lake on a daily basis
Declined	None	T205932 Create table of pre-Event Bus change tags
Resolved	JAllemandou	T205940 Add change tag tables to monthly mediawiki_history sqoop

Event Timeline

nshahquinn-wmf claimed this task.Jul 4 2018, 12:54 AM

nshahquinn-wmf triaged this task as High priority.

nshahquinn-wmf created this task.

nshahquinn-wmf moved this task from Triage to Next Up on the Product-Analytics board.

nshahquinn-wmf updated the task description. (Show Details)Jul 4 2018, 12:58 AM

nshahquinn-wmf added a subscriber: • chelsyx.Jul 4 2018, 1:01 AM

nshahquinn-wmf moved this task from Next Up to Doing on the Product-Analytics board.Jul 5 2018, 8:02 PM

MNeisler subscribed.Jul 10 2018, 8:27 PM

nshahquinn-wmf added a subtask: T201062: Load change tags into the Analytics Data Lake on a daily basis.Aug 2 2018, 6:12 PM

• Tbayer subscribed.Aug 7 2018, 7:06 PM

nshahquinn-wmf added a subtask: T201501: Develop a framework for measuring user retention.Aug 8 2018, 10:55 AM

nshahquinn-wmf updated the task description. (Show Details)Aug 8 2018, 3:02 PM

nshahquinn-wmf updated the task description. (Show Details)Aug 8 2018, 7:08 PM

nshahquinn-wmf updated the task description. (Show Details)Aug 30 2018, 11:02 PM

Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptAug 30 2018, 11:02 PM

nshahquinn-wmf removed a subtask: T201501: Develop a framework for measuring user retention.Aug 30 2018, 11:03 PM

nshahquinn-wmf moved this task from Doing to Next Up on the Product-Analytics board.Sep 7 2018, 6:51 PM

nshahquinn-wmf closed subtask T201062: Load change tags into the Analytics Data Lake on a daily basis as Resolved.Oct 1 2018, 7:45 PM

nshahquinn-wmf updated the task description. (Show Details)Oct 1 2018, 9:52 PM

• JKatzWMF closed subtask T198777: Pilot new editor retention and mobile new editor retention calculations as Declined.Oct 4 2018, 10:56 PM

• JKatzWMF moved this task from Next Up to Epics on the Product-Analytics board.

@Neil_P._Quinn_WMF to close out this task and file separate follow-up tasks/requests as needed (e.g. In mediawiki_history, we'd calculate reverts using the convenient is_reverted field. How do we do it when we simply have a table of revisions with their hashes?)

As Kate said, closing this because this parent task doesn't have any value. I've separated out the reverts issue as T216297.

kzimmerman moved this task from Epics to Blocked on the Product-Analytics board.Aug 9 2019, 5:55 PM

nshahquinn-wmf closed subtask T205932: Create table of pre-Event Bus change tags as Declined.Apr 10 2020, 5:17 PM

Set up data infrastructure for program metricsClosed, DeclinedPublicActions