Page MenuHomePhabricator

Infrastructure for interventions impacting editing metrics
Closed, DeclinedPublic

Description

A lot of Audiences teams' FY18-19 annual plans rely on editing metrics (e.g. new editor acquisition, retention of new/existing editors, edit revert rates). In cases where possible, those teams would like the ability to do A/B tests of new features meant to change those metrics to evaluate their efficacy before releasing them.

So while we have prior work, methodologies, and technologies in place for doing experiments of changes to reading experiences (e.g. Page Previews), we currently lack the infrastructure and data pipeline for:

  • sampling contributors to enable features at random and on specific platforms, perhaps in cohorts
  • tracking which contributors are using what experimental feature
  • tracking which contributions were made with an experimental feature

Therefore, this might very well be a multi-team/departmental endeavor.

Event Timeline

To clarify just in case: A/B testing of editing features and/or the sampling of contributors has been done before, e.g. by the old Growth team (example) or in the Teahouse study. In some sense it is actually easier than for readers because (logged-in) editors have persistent, public user IDs. I understand the current task is about building a more general, easy to use infrastructure.

kzimmerman triaged this task as Medium priority.Jun 28 2019, 6:34 PM
kzimmerman added a project: Better Use Of Data.
kzimmerman subscribed.

This falls within the scope of work we have planned for FY 19-20, tagging with Better Use of Data. It may be adjusted/consolidated later.

mpopov renamed this task from [EPIC] Infrastructure for interventions impacting editing metrics to Infrastructure for interventions impacting editing metrics.Dec 10 2019, 9:03 PM
mpopov removed a project: Epic.
mpopov removed a subscriber: Tbayer.
mpopov added a subscriber: sdkim.

@jlinehan @sdkim: Updating ownership, feel free to decline/re-prioritize as needed

jlinehan lowered the priority of this task from Medium to Lowest.Aug 24 2020, 3:18 PM
jlinehan moved this task from Inbox to Task Backlog on the Product-Data-Infrastructure board.