Page MenuHomePhabricator

Establish baseline for Q2 OKR and conduct preliminary Talk page research
Closed, ResolvedPublic

Description

Background:
The Android team is working to enhance the messaging interface. The OKRs we originally established are as follows:
Objective: Target users have a more intuitive and enhanced communication interface to reduce barriers for productively collaborating with other Wikipedians to increase confidence while improving wikiprojects
Key results (Overall increase in Talkpage edits): Talk Namespace edits made through Android app and not reverted within 48 hours increases by 10% 30 days after release of interventions
Key results( Penetration Increase amongst active editors): Of the people who publish an edit to a wiki, across namespaces, in a month, we see an 10% increase in the percentage of people who publish an edit to a talk page that is not reverted within 48 hours.
Key Result: 2% Decrease in abandonment rate amongst target users on talk pages

The questions we will also monitor is:

  • Do we see a change in time spent/session length
  • Are there improvements throughout the funnel and are there differences when compared to Mobile Web
  • Have our changes impacted user talk pages vs article talk pages in different ways?

Task:

  • Share with the Android team the current baseline for the Key Results listed above or establish that the data is too difficult to collect so that modifications to the OKRs can be modified
  • Share with the Android team the answer to the following questions
    • What do we know about the Android app talk page usage presently? Take into consideration questions on T293114
    • What is the # of edits to the talk page namespace comparative to number of editors/general edits in app for native talk pages VS # of edits to talk page namespace on mobile web comparative to the number of editors/general edits on mobile web
    • What is the bounce rate/abandonment for talk pages by platform? Specifically, did someone attempt to edit a talk page and failed to / didn't click publish. How does this bounce rate compare to mobile web?

Additional Detail

  • Any interesting insights about talk pages once the native talk pages were released
  • Feel free to be agile, by pulling what information is easier to get (ex. by project) and working on the more complicated parts later and sharing it once you have it
  • For the bounce/abandonment rate

Event Timeline

JTannerWMF renamed this task from Establish baseline Q2 OKR to Establish baseline for Q2 OKR and conduct preliminary Talk page research .Nov 8 2021, 11:46 PM
JTannerWMF updated the task description. (Show Details)

Baseline data for target user groups, mobile web Talk page edits, Talk page bounce rate by platform.

We modified the OKRs.

@SNowick_WMF will make modifications to the spreadsheet an update this task with

  • Sum of Edits from talk page namespace divided by "Sum of Unique Editors"
  • Definition of event actions, specifically surrounding the abandonment rate funnel in EditAttemptStep

@JTannerWMF - Updated unique counts and edit ratios for edits/editor by Namespace for each target group - tabs added to main Android, Mobile Web Talk Page Edit/Editors T294971 dataset. This gives 60 days of baseline data in preparation for analyzing engagement after Talk page design/function changes are made.

Added metrics for Target Group Talk page Edit Completion Rates to deck. Link to data.

Note: This data is outdated, see https://phabricator.wikimedia.org/T301023#7784211 for correct stats and analysis.

Talk Page Edit Completion Rate by Platform Data (90 day date range, 2021-11-01 - 2022-01-30)

Article Talk Page

Platform:StartSave Success% Complete
Desktop92974012361813.3 %
Mobile Web20146145742.3 %
Android App5968149325 %

User Talk Page

Platform:StartSave Success% Complete
Desktop60163516291227.1 %
Mobile Web37255685018.4 %
Android App8318227027.3 %

Combined Talk Pages

Platform:StartSave Success% Complete
Desktop153137528653018.7 %
Mobile Web238716114244.8 %
Android App14286376326.3 %

Data Sources:
Desktop, Mobile Web: EditAttemptStep
Android App: MobileWikiAppEdit
Data

Holding this one until we have an idea of our funnel comparative to other platforms

JTannerWMF raised the priority of this task from Medium to High.Mar 10 2022, 1:59 PM

Moving this task to @SNowick_WMF column. Based on our touch base she will add three new KRs related to:

  • Key Result: % increase of engagement with talk pages and of active editors
  • Key Result: Discoverability metric
  • Key Result: Active users Returning to talk pages

and add the relevant baseline data

KR Definitions:

Engagement

  • Key Result: 2% increase of user engagement with talk pages Phase 1
  • 2% increase above that for Phase 2

Engagement is defined as frequency (count) of Talk Page actions per unique users, and by Target groups, over time.
Also measures event counts relevant to overall DAU and user groups as average percent of DAU. ( See Android DAU Data - contains geodata, please request access if needed.)
Phase 1 will be post Talk Page redesign release and Phase 2 will be post Talk page bubble/sidebar release.

Baseline is current data for all users and by Target groups. Date range is 2022-02-01 - 2022-03-11.

Current Baseline:

User GroupAvg Events/Unique User% of Total Event CountsGroup as % of DAU (average daily active users by group/average Android daily active users)
All Talk Page Users1.78100.00%0.66%
hiwiki1.450.56%0.13%
idwiki1.681.14%0.40%
jawiki1.812.06%2.69%
India, Nigeria enwiki1.508.86%6.44%
Mali, DR Congo,Morocco,Egypt arwiki, frwiki1.500.83%0.40%

Retention

  • Key Result: 1% increase in 3 day and 14 day Talk page user Retention

Of users who showed activity in Talk pages, what percent return to Talk pages and how frequently do they return.
Baseline is current data for all users and by Target groups. Date range is 2022-02-01 - 2022-03-11/2022-02-28 for 14 Day Retention

Group3 Day Retention14 Day Retention
All Android Users - All Pages (opt-in to share data)63.67%76.05%
All Talk Page Users9.01%13.61%
hiwiki Talk Page Users4.59%3.08%
idwiki Talk Page Users9.29%12.03%
jawiki Talk Page Users11.38%16.41%
India, Nigeria enwiki Talk Page Users5.23%6.11%
Mali, DR Congo, Morocco, Egypt arwiki, frwiki Talk Page Users6.56%8.73%

Discoverability

  • Key Result: Discoverability metric TBD

Are we increasing Talk page activity for new users? Increase in first-time Talk page visits over time.
There are a few options for this metric I want to explore w my team to make sure we are using the best way to gauge this.
Note that we can use the current % of DAU active on Talk Pages data as an initial Baseline as well, increase in that percentage will indicate more users are using Talk Pages overall.

Discoverability
Baseline KR is average daily percent of user population that is new to Talk pages (not seen using Talk since we started measuring visits). Records of active Talk page users can be recorded prior to each new release, so we will always be able to detect discovery and new users. Will post Baseline average here when we have accumulated an acceptable sample size in the next week.

Time Spent
We are including this baseline for posterity, with all the caveats that measuring user time in mobile is unreliable for multiple reasons. The current baseline for date range 2022-03-01 - 2022-03-27: Data

UsersAvg. Time Spent (seconds) per Unique/Daily
All users58.35
Anon users47.13
Logged In users88.00

Discoverability
We have baseline data on returning vs new users as percent of population based on a snapshot taken in late 2022-03, this snapshot will be updated just prior to the release of new Talk page features so this metric should be considered a temporary stand-in for values we will update with each new release.

The average distribution returning Talk page users to of new users for the date range 2022-03-23 - 2022-04-20:

Avg. known/returning unique usersAvg. new unique users
30%70%

Data