Page MenuHomePhabricator

[EPIC] Measuring the impact of exposing talk pages to mobile web anons
Open, HighPublic

Description

Background

T293946: Enable talk for mobile users on enwiki will make it so people who are logged out visiting English Wikipedia on a mobile device will see a link to talk pages on the articles they're viewing.

This task is about identifying what metrics we would like to calculate in order to evaluate the impacts said change is having on talk pages and the people who depend on them.

Metrics definition

Questions we want to answer

  • How does this change affect the amount of vandalism and other non-relevant content on talk pages?
  • How does this change affect the communication between anonymous and logged-in editors?
    • Does this change affect the overall number of IP editors blocked from editing?
    • Does this change affect the number of unreverted edits made by IP editors (split by namespace)?
  • Does exposing readers to talk pages decrease their overall trust in Wikipedia?
  • Do readers understand the purpose of the talk page and how to interact with it?
    • Do they spend any substantial amount of time on talk pages (definition of substantial TBD)?
    • Do they leave the site entirely after visiting the talk page?

From initial description:
This section will contain the metrics we would like to calculate and what – if any – new instrumentation would need to be implemented to do so.

IDMetricNew Instrumentation Required (Y/N)Notes

Done

Web team Instrumentation Spec

Related Objects

Event Timeline

ovasileva triaged this task as High priority.
ovasileva updated the task description. (Show Details)
ovasileva added a project: Readers-Web-Backlog.
ovasileva moved this task from Incoming to Analyst Consultation on the Readers-Web-Backlog board.
ovasileva added a subscriber: jwang.

Hi @ovasileva , in the Metrics collection, the metric to answer below question is vague. "Trust in Wikipedia" is not a metric can be measured by quantitative data.

Does exposing readers to talk pages decrease their overall trust in Wikipedia?

We can discuss it in our meeting on Wednesday.

Hi @ovasileva , in the Metrics collection, the metric to answer below question is vague. "Trust in Wikipedia" is not a metric can be measured by quantitative data.

Does exposing readers to talk pages decrease their overall trust in Wikipedia?

We can discuss it in our meeting on Wednesday.

@jwang - this one we were planning to measure through a quicksurvey - will make it clearer in the list!

The current MobileWikiAppTalk schema has metrics for:
logged-in status
namespace
time_spent

We can explore adding a field for page name, will discuss with @Sharvaniharan in our next meeting.

Will talk page be enabled on both mobile web and mobile app, or just mobile web? @ovasileva

Hi @ovasileva, I have updated instrumentation spec based on our discussion. In the tab of new_in_schema, a few details needs PM and R&D's decisions, marked in Note column.

Update from the Editing Team
@MNeisler is reviewing the Metrics collection sheet to determine:

  1. What – if any – additional metrics we ought to consider gathering to answer the questions in the task description's Questions we want to answer section in conjunction
  2. Whether the instrumentation necessary to report on the Metrics listed within the Metrics collection sheet is currently implemented

Quick update on my review:

The proposed metrics look good. I added any additional thoughts and comments in the Metrics collection sheet but wanted to highlight one potential issue:

Whether the instrumentation necessary to report on the Metrics listed within the Metrics collection sheet is currently implemented

I remembered we have an open ticket (T249944) re all WikiEditor edits tagged as coming from desktop in EditAttemptStep.  This may impact our ability to reliably distinguish any talk page anon Wikieditor edits coming from desktop vs mobile web for metrics calculated using EditAttemptStep or the new talk_page_edit schema, which I believe is currently only the following metric:

"What percentage of new topics that are created by mobile anons on article talk pages get a response from anyone?"

If confirmed this impacts mobile web talk page edits, we might want to consider re-prioritizing T249944.
CC @ppelberg

Quick update on my review:

The proposed metrics look good. I added any additional thoughts and comments in the Metrics collection sheet but wanted to highlight one potential issue:

Whether the instrumentation necessary to report on the Metrics listed within the Metrics collection sheet is currently implemented

I remembered we have an open ticket (T249944) re all WikiEditor edits tagged as coming from desktop in EditAttemptStep.  This may impact our ability to reliably distinguish any talk page anon Wikieditor edits coming from desktop vs mobile web for metrics calculated using EditAttemptStep or the new talk_page_edit schema, which I believe is currently only the following metric:

"What percentage of new topics that are created by mobile anons on article talk pages get a response from anyone?"

If confirmed this impacts mobile web talk page edits, we might want to consider re-prioritizing T249944.
CC @ppelberg

@MNeisler - is it possible to distinguish by skin here? Generally when we're talking about the mobile site here, we're really talking about minerva usage on mobile (and similarly the changes will show on minerva in desktop)

That's approximately what the current meaning of mobile/desktop is in the schema -- if you're in the MobileFrontend experience you get tagged as mobile, otherwise you get tagged as desktop.

Quick update on my review:

The proposed metrics look good. I added any additional thoughts and comments in the Metrics collection sheet but wanted to highlight one potential issue:

Whether the instrumentation necessary to report on the Metrics listed within the Metrics collection sheet is currently implemented

I remembered we have an open ticket (T249944) re all WikiEditor edits tagged as coming from desktop in EditAttemptStep.  This may impact our ability to reliably distinguish any talk page anon Wikieditor edits coming from desktop vs mobile web for metrics calculated using EditAttemptStep or the new talk_page_edit schema, which I believe is currently only the following metric:

"What percentage of new topics that are created by mobile anons on article talk pages get a response from anyone?"

If confirmed this impacts mobile web talk page edits, we might want to consider re-prioritizing T249944.
CC @ppelberg

Notes from this morning's conversation

  1. T249944 does impact mobile web talk page edits in so far as until T249944 is fixed:
    • Edits made by people who are using the desktop site on a mobile device will be logged as having been published from platform = desktop
    • Edits made by people who are using the mobile site on a mobile device with Javascript disabled will be logged as having been published from platform = desktop
  2. "1." means that if we proceed without fixing T249944 when answering "What percentage of new topics that are created by mobile anons on article talk pages get a response from anyone?", we will be excluding some number [i] "new topics that are created by mobile anons" from this calculation

Next steps

  • @ppelberg to talk with @MNeisler + @DLynch to learn the level of effort required with fixing T249944
  • @ppelberg to decide whether fixing T249944 ought to block us answering "What percentage of new topics that are created by mobile anons on article talk pages get a response from anyone?"

i. Note: we do not currently know how many edits we may potentially be excluding because we do not currently have a way of tracking edits made using the desktop site where platform = mobile (T288972)

To elaborate on my previous comment here:

A little work would need to be done for this to support the change to platform. We'd either want to upstream the mobile device detector from mobilefrontend into mediawiki core so it could be used in multiple places, or agree on some other consistent metric for logging this that instrumentation could consistently use -- one might already exist in some other eventlogging usage. We don't have to use the same thing that triggers mobilefrontend.

We'd also need to change various schema so there's an explicit integration for mobilefrontend.

Oh, specifically, we need a definition of platform that can be consistently applied from client- and server-side logging code.

Next steps

@DLynch, @MNeisler, and I talked about this.

T294503#7493119 documents the approach for fixing T249944 David proposed and Megan, and the wider Product Analytics Team, supports.

  • @ppelberg to decide whether fixing T249944 ought to block us answering "What percentage of new topics that are created by mobile anons on article talk pages get a response from anyone?"

While the Editing Team will prioritize work on fixing T249944 by way of implementing what David proposed in T294503#7493119 [i], the analysis being planned in this ticket does NOT need to be blocked on T249944 being resolved. [ii]


i. I've updated T249944's task description with the approach David articulated in T294503#7493119
ii. Thinking: we are assuming that exposing the "talk" tab to mobile anons will cause a nominal increase in the number of new topics mobile anons use the desktop site to create considering the amount of awareness and number of steps required to do so: 1) Click "Talk", 2) Bypass the Add discussion button at the top of the page, 3) Scroll to the bottom of the talk page, 4) Click the Desktop link, 5) Click Edit, 6) etc.

ovasileva renamed this task from [SPIKE] How might we measure the impact of exposing talk pages to mobile anons? to [EPIC] How might we measure the impact of exposing talk pages to mobile anons?.Nov 18 2021, 10:54 AM
ovasileva renamed this task from [EPIC] How might we measure the impact of exposing talk pages to mobile anons? to [EPIC] Measuring the impact of exposing talk pages to mobile anons?.
ovasileva renamed this task from [EPIC] Measuring the impact of exposing talk pages to mobile anons? to [EPIC] Measuring the impact of exposing talk pages to mobile anons.
ovasileva removed ppelberg as the assignee of this task.
ovasileva added a project: Epic.
ldelench_wmf renamed this task from [EPIC] Measuring the impact of exposing talk pages to mobile anons to [EPIC] Measuring the impact of exposing talk pages to mobile web anons.Nov 22 2021, 5:07 PM
ldelench_wmf moved this task from Triage to Epics on the Product-Analytics board.

Hey is it possible/on your roadmap to decouple the mobile. Specifically we can't see a difference between Mobile Web and the Apps

Hey is it possible/on your roadmap to decouple the mobile. Specifically we can't see a difference between Mobile Web and the Apps

The schema already has an integration available for app -- presumably the apps should be submitting their events using it, and that would let us see the difference between mobile web and apps? (Well, presumably it should be even more granular such that we can see the difference between android and ios, but...)

Following @DLynch's comment - the apps teams could add an android app or ios app flag to edit events made in app to populate the EditAttemptStep integration column, but where would we need to add that to insure it's captured downstream? We would also need to have android app and ios app made allowable by integration if it is an enum data-type (not sure is that matters in the MEP system but it did in Legacy). Adding @Sharvaniharan and @Dbrant as they will be implementing any code we decide to add. I will follow up with the iOS team as well once this is ironed out.

Following up on this - by default, all edits made in the Android app have an android app edit tag, iOS edits have an ios app edit tag - how do we arrange for the EdittAttemptStep to populate the integration column with these tags? This would allow us to definitively separate out edits made on mobile apps. @DLynch can we discuss getting this implemented?

@DLynch can we discuss getting this implemented?

hi @SNowick_WMF – yep. In fact, it looks like we have a meeting scheduled for tomorrow, 14-Dec, at 20:30 UTC, titled Editing + Android metrics for our two teams to talk about this.