[Impact Analysis] Evaluate Impact of Mobile Reply and New Discussion Tools
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ppelberg
	Dec 20 2021, 11:37 PM

Description

T282638 will introduce the Reply and New Discussion Tools on mobile at an initial set of partner wikis.

This task represents the work with conducting an impact analysis to evaluate what impact introducing the Reply and New Discussion Tools on mobile and setting the Read as wiki page view as the default has had on:

Peoples' likelihood to successfully publish new talk page topics and comments they initiate using the mobile site
Peoples' likelihood to publish edits to talk pages on mobile that are disruptive to other volunteers.

Analysis Scope

The data analyzed as part of this ticket should be limited to the two weeks before and after T298221 is resolved.

Decision to be made

This impact analysis will help us make the following decision: Do any adjustment need to be made to the mobile Reply and New Discussion Tools before making them available to more people by way of the A/B test we are planning in T298062?

Research questions

After the mobile Reply and New Topic Tools were introduced, how – if at all – did the likelihood that people would successfully publish the new talk page topics and comments they started drafting change?
After the mobile Reply and New Topic Tools were introduced, how – if at all – did the likelihood that people would publish talk page edits that are reverted within 48 hours of being published?.

Decision Matrix

ID	Scenario	Plan of action

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T278588 Mobile talk page improvements
Resolved	None	T298055 [RELEASE TICKET] Offer Mobile Usability Improvements and Topic Subscriptions at Partner Wikis
Resolved	• ppelberg	T298058 [Impact Analysis] Evaluate Impact of Mobile Reply and New Discussion Tools

Event Timeline

• ppelberg created this task.Dec 20 2021, 11:37 PM

• ppelberg edited projects, added Editing-team (Tracking); removed Editing-team.

• ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.

• ppelberg added a subscriber: MNeisler.

https://phabricator.wikimedia.org/auth/

• BilalShirwani removed projects: Editing-team (Tracking), DiscussionTools.Dec 20 2021, 11:48 PM

• BilalShirwani removed subscribers: MNeisler, Parkywiki, Pelagic and 5 others.

• BilalShirwani merged tasks: T297999: Coordinate links overlap with image, T297997: unchecked response dict access in pywikibot , T297996: Navigating to Special:Preferences starts an infinite loop of redirects, T297993: Enable prepared PostgreSQL statements, T297987: Test, T297955: Upgrade php image to stretch/php74-fpm, T297962: Deploy improved title-parsing code, T298006: wrong encoding recognised by reflinks.py, T298018: purgeExpiredMentorStatus.php does not operate correctly when number of rows to delete is higher than batch size, T298015: Modify interface message "Wikimedia-globalblocking-ipblocked-range", T298059: Hard deprecate and remove the global function wfWikiID(), T298057: Account creation: JOSA landing page, T298056: AbuseFilter should provide a way to get WebRequest in custom variable hooks, T298055: [RELEASE TICKET] Offer Mobile Usability Improvements and Topic Subscriptions at Partner Wikis, T298054: Update argument validation for built-in list functions to exclude Z10s, T298053: Add a phan job to gated extensions/skins and core testing each change with all of them, T298051: Content from a template duplicated in a DiscussionTools edit (it.wikipedia), T298050: MediaWiki Won't Connect to SQLite in Docker, T298049: Ensure User-Defined Z4s Always Produce UserDefinedTypeKeys, T298048: Growth features is making a lot of db queries in every pageview regardless of user's status, T297942: Specific PDF on Commons has no image thumbnails, dimensions shown as 0x0 pixels, T297943: Use of funds overlay does not display on click, T297944: Set up regular-repairs for AQS cassandra cluster tables, T297945: [survey] Investigate existing features to allow more conveniently submitting votes, T297947: Purge expired blocks older than 90 days, T297948: Allow Wikimedia Maps usage on wikijournal.org, T297949: Can't render page listed in $wgWhitelistReadRegexp, T297951: Web2Cit should reject top-level domains, T297952: deploy quicksurvey for editors on bnwiki (for Growth IP editors research), T297953: Sticky header: LTR Username text and icon looks out of place on sticky header user menu on RTL wikis, T297956: Dashboard/Documents -> while execution endpoints from the dashboard the CORS policy error appears , T297957: IPInfo: Help popup position makes it unclear what its attached to, T297958: Cross Environment Token Usage, T297959: thanos-be hosts filing up root filesystem with logs, T297960: Two lines after zoom-in zoom-out, T297961: Add Proxy Information to infobox [L], T297963: MultilineTextInputWidget with autosize=true grows too early, T297964: TypeError: Cannot read properties of undefined (reading 'length'), T297965: Logo images for Tate article does not appear correctly in black theme (images with black on transparent), T297966: Auto topic subscription should be enabled by default on 3rd party installs, T297967: Figure out a better solution for parsing & handling URL params for Special:MediaSearch in PHP, T297968: Allow Wikimedia Maps usage on bbcrewind.co.uk, T297969: wikilambda_function_call-not-connected when installing fresh with docker compose v2, T297970: Error when using Translate extension to translate {{DISPLAYTITLE:}} contents, T297971: Can't npm install without deleting package-lock, T297972: Duplicated Category: wikilinks created by Special:ContentTranslation on viwiki, T297973: Feature request - Improve readability of blue link text on black/dark themes, T297974: "Title" is unclear, and should probably say "Subject" or "Topic" instead, T297975: "Password" inputbox still appear if "createaccountmail" checkbox is selected, T297977: [Notifications] Add selection state to notifications cells, T297976: VisualEditor's LanguageVariantNode didn't parse annotations inside to respective nodes, T297978: Switching from legacy new section editing in 2017WTE to New Discussion Tool via the "hint" opens a new tab, T297979: Add new watchlist icon, and amend related 'recentChanges' & 'userContributions' icon for DSG, Codex & OOUI, T297980: Reply tool sometimes breaks up comments into lots of lines, T297981: Apply for Erasmus+ for the Grand Tour of Wikimedia, T297982: Apply for Erasmus+ accreditation, T297983: Research Showcase March 2022, T297984: Media html read view considerations, Restricted Task, T297986: [Tracking task] Pair with brian king on various operational tasks, T297988: Share Topic Container design approaches with partner wikis and populations, T297989: Function with composition sometimes returns "no valid Z8K4 error", T297990: Clarify how the "Enable quick topic adding" setting affects creating new talk pages (separate setting?), T297991: Create semgrep initial tool ci template, T297992: Backend server is not starting in dockerized image, T297994: pywikibot should not retry if that doesn't make sense, T297995: Remove authentication from Wikimedia Commons Query Services (WCQS), T297998: Collect statistics about references on Wikis, T298000: Disregard Wikimedia Foundation Wiki from block criterion, T298001: Wikidata claims editor is broken after December 2021 update on non-ES6 browsers, T298002: Visual diff crashes on table in list, T298003: Some fields with a local exception displayed wrongly in Special:GlobalPreferences, T298004: Pandora has changed format, T298005: Upgrade Toolforge Kubernetes to version 1.23, T298008: shellbox-pipeline-test jobs fail because they use php-ast 0.1.6, T298007: Add idwiki to WikiWho, T298009: Localisation update deleted many i18n files for Collection extension, T298010: Querying ProcseeBot's user logs results in internal_api_error_DBQueryError, T298011: JPEG image is reported with the wrong mime-type application/octet-stream, T298012: List type change is described once for each list item, T298013: Please add Version to ExtensionDependencyError, T298016: Create OgvJS project as subproject of TimedMediaHandler, T298017: Add functionality to the Audio/Video filepage to generate links with start/stop query parameters, T298020: Dropdown component should not emit event when user clicks the selected option, T298021: Integrate Bullseye 11.2 point update, T298022: Solutions to avoid T298009 from happening again, T298023: ProofreadPage: Source tab should not appear on the Main Page, T298024: Create contact page form on Meta-Wiki for Chapter/Thematic org status application, T298025: Allow passing membership fee amount to form, T298026: Research Showcase May 2022, T298027: [INVESTIGATION] Evaluate speed of key interactions on Wikibase.dev, T298029: Enable Content Translation beta feature for a user when accessing a Section Translation entry point on mobile, T298028: Move old wikipedia.de banners to subdirectory, T298030: Visual Editor hangs with Template:Graph:Lines, T298031: Mentees assigned to others mentors are visible in my recent changes list, T298032: Surface missing languages in current mobile language selector to access Section Translation, T298033: Update bnwikivoyage wordmark logo, T298034: Tester page seems broken, T298035: Prepare OOUI for Google Chrome userAgent deprecation, T298036: OpenSearch coordinate map view thinks it is broken, T298037: Exception thrown and comment not highlighted when replying to non-subscribable sections, T298039: Migrate TwoColConflict usage of objectcache away from the table, T298040: replace existing Tooltip with CODEX version, T298041: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured", T298042: evaluate & drop cloud NAT exceptions for APT repositories, T298043: Revisit how CI works for released branches, T298044: Prepare ResourceLoader for Google Chrome userAgent deprecation, T298045: Notice: Undefined index: href in /../w/skins/Strapping/StrappingTemplate.php on line 596, T298046: Provide a way to run Parsoid against "latest git HEAD of mediawiki-vendor" during round trip testing on scandium, T298047: Performance Team onboarding for tstarling.Dec 20 2021, 11:55 PM

• BilalShirwani added subscribers: Edtadros, WolfgangFahl, Esanders and 196 others.

Legoktm changed the edit policy from "All Users" to "Administrators".Dec 21 2021, 12:00 AM

Legoktm unsubscribed.Dec 21 2021, 12:06 AM

MusikAnimal unsubscribed.Dec 21 2021, 12:06 AM

Legoktm added projects: Editing-team (Tracking), DiscussionTools.Dec 21 2021, 12:07 AM

Legoktm removed subscribers: dr0ptp4kt, jhathaway, Andrew and 195 others.

Legoktm changed the edit policy from "Administrators" to "All Users".Dec 21 2021, 12:24 AM

@ppelberg all the vandalism has been cleaned up, but I reset the entire subscriber list so you should re-CC anyone who you wanted to see this.

Aklapper added subscribers: MNeisler, Parkywiki, Pelagic and 4 others.Dec 21 2021, 1:30 AM

In T298058#7582062, @Legoktm wrote:

@ppelberg all the vandalism has been cleaned up, but I reset the entire subscriber list so you should re-CC anyone who you wanted to see this.

I appreciate you addressing this issue as quickly as you did – thank you, @Legoktm and thank you, @Aklapper for restoring the subscriber list.

• ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.Dec 22 2021, 4:07 PM

• ppelberg updated the task description. (Show Details)Dec 23 2021, 12:36 AM

• ppelberg mentioned this in T298065: [Impact Analysis] Evaluate Impact of mobile Topic Subscriptions + Usability Improvements.Dec 23 2021, 1:00 AM

• ppelberg mentioned this in T297448: [SPIKE] Review approach to improving mobile talk pages.Dec 23 2021, 1:20 AM

• ppelberg mentioned this in T298271: [SPIKE] Determine mobile talk page usage volume needs.Dec 23 2021, 9:17 PM

• ppelberg mentioned this in T282638: [RELEASE TICKET] Offer Mobile Reply and New Discussion Tools at Partner Wikis.Feb 5 2022, 12:04 AM

• ppelberg mentioned this in T301026: Create instrumentation spec for mobile DiscussionTools.Feb 5 2022, 2:09 AM

• ppelberg mentioned this in T302108: Ensure logging is in place to compare MobileFrontend and DiscussionTools new topic and new comment completion rates.Feb 18 2022, 7:27 PM

• ppelberg assigned this task to MNeisler.Jul 20 2022, 4:19 PM

• ppelberg added a project: Product-Analytics.

• ppelberg mentioned this in T312685: [SPIKE] Decide whether additional config is necessary to isolate mobile and desktop Usability Improvement deployments.Jul 20 2022, 11:05 PM

MNeisler triaged this task as Medium priority.Jul 22 2022, 6:08 PM

MNeisler moved this task from Triage to Current Quarter on the Product-Analytics board.

• ppelberg added a project: Goal.Jul 27 2022, 9:37 PM

Meta: I've updated the task description to reflect the scope that @MNeisler and I came to define in the meeting we had offline today.

MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Aug 3 2022, 8:05 PM

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Aug 11 2022, 7:34 PM

I completed a pre and post-deployment analysis to determine the impact of introducing Reply and New Discussion Tools on mobile at an initial set of partner wikis. See summary below and full report for details on how these values were calculated and additional breakdowns.

Data: Reviewed data includes talk page events logged two week prior and post deployment of the feature on 29 June 2022. (Timeframe: 14 June through 13 July 2022). Below analysis compares talk page edits on the DT-enhanced version of mobile web talk pages available post-deployment to both the existing, Read as wiki page and MobileFrontend talk overlay available pre-deployment (Note: Ability to track MobileFrontend edit workflows limited pending T302108)

Peoples' likelihood to successfully publish new talk page topics and comments they initiate using the mobile site

Overall

There was a 53.4% (12 percentage points) increase in talk page edit completion rate overall.

By Logged-In Status

edit_completes_overall_combined_byanon.png (2×4 px, 192 KB)

This increase was seen for both logged-in and logged-out users. For logged-in users, there was a 51.4% increase (15 percentage points). For logged-out users, there was a significant increase - edit completion rate tripled (an increase by 9.9 percentage points).

Peoples' likelihood to publish edits to talk pages on mobile that are disruptive to other volunteers.

Overall

By Logged In Status

reverts_pre_post_byanon.png (2×4 px, 174 KB)

We observed a decrease in reverted edits for both logged-in and logged-out users. There was an 5.6% decrease for logged-in users and a 10.2% decrease for logged-out users.

Peoples' likelihood to engage on the talk page by either clicking the add a topic button or section heading

There was a 19% increase in mobile talk page engagement post-deployment

Percent of talk page views where the users engages on the page by either clicking the add a topic button or section heading

Pre or post deployment	Percent of talk page views
post	34.67%
pre	29.13%

Overall mobile web edits

Did a quick check of the total daily overall mobile web edits on partner wikis and confirmed there were no significant changes pre or post deployment in the total number of edits. See Superset Chart

Instrumentation Notes and Caveats:

At the time of deployment, all discussiontool related events on mobile were incorrectly tagged as desktop. A patch was submitted to fix this for future analyses. For the purpose of this analysis, I relied on the pagetoken to join EditAttemptData with MobileWebUIActions to determine mobile DT edits.
Not all components of the Mobile Frontend edit workflow described in T302108 are currently instrumented in MobileWebUIActions. Pending the resolution of that task, data for this analysis was limited to Read as Wiki completion rates on mobile web and clicks to the add topic button on the mobile talk page overlay. I also looked at engagement (percent of views that lead to a click) and overall published mobile edits as a way to help more fully understand changes in interaction on mobile talk pages.

@ppelberg - Reassigning to you for review and sign-off. Let me know if you have any questions.

nshahquinn-wmf awarded a token.Aug 30 2022, 12:42 AM

@ppelberg

I've updated the report with edit attempt completion rate metrics broken down by experience level. See chart below:

edit_completes_overall_combined_byexp.png (2×4 px, 187 KB)

Similar to trends we've seen with the DiscussionTool deployments on desktop, there was a significant impact on completion rate specifically for Junior Contributors with under 100 edits and contributors with between 100-499 edits. Senior Contributors saw a slight decrease. It is important to note that since only a pre and post-deployment analysis was completed, it's possible some other effects such as seasonality effects may account for some of these changes; however, given the significant observed increase, it is very likely that deployment of the DiscussionTools on mobile contributed to these changes.

Let me know if you have any questions.

Codebase

In T298058#8206832, @MNeisler wrote:

@ppelberg

I've updated the report with edit attempt completion rate metrics broken down by experience level. See chart below...

This looks great – thank you, @MNeisler. I've published the results on-wiki here: https://w.wiki/5ei9.

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptSep 2 2022, 1:29 AM

	F35505786: edit_completes_overall_combined_byexp.png
	Sep 1 2022, 6:51 PM

	F35488149: edit_completes_overall_combined_byanon.png
	Aug 24 2022, 6:23 PM

	F35488076: edit_completes_overall_combined.png
	Aug 24 2022, 6:23 PM

[Impact Analysis] Evaluate Impact of Mobile Reply and New Discussion ToolsClosed, ResolvedPublicActions