Page MenuHomePhabricator
Create Task
Maniphest T327528

Pull data about Article Description Suggested Editors
Closed, ResolvedPublic
Actions

Description

Background
The team is experimenting with suggesting machine generated article descriptions to assist users in creating short descriptions. We need to identify our target test languages and regions. The test audience needs to be large enough to understand the integrity of the algorithm and usefulness of the feature.

Task

  • Pull baseline data about editors as defined in our Shallow Deck
  • Answer: What is the minimum number of annotations and participants we need to hit for article descriptions to validate the hypothesis?

Important Additional Context

  • The number of participants in a similar experience and languages included (Reference)
  • There is a constraint that the feature can only be tested in the mBART 25 (Reference)

Related Objects
Search...

Event Timeline

JTannerWMF triaged this task as High priority.Jan 20 2023, 5:00 PM
JTannerWMF created this task.
JTannerWMF moved this task from Epics in Progress to Needs Analytics on the Wikipedia-Android-App-Backlog (Android Release - FY2023-24) board.
SNowick_WMF renamed this task from Pull data about Article Description Suggesed Editors to Pull data about Article Description Suggested Editors .Jan 20 2023, 7:56 PM
mpopov moved this task from Triage to Kanban on the Product-Analytics board.Jan 24 2023, 6:09 PM
mpopov edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
JTannerWMF updated the task description. (Show Details)Jan 31 2023, 12:17 AM
SNowick_WMF added a comment.Jan 31 2023, 6:13 PM

Data is ready for 2022-12, will add 2023-01 data when mediawiki_history is rolled up.

SNowick_WMF added a comment.EditedFeb 3 2023, 8:16 PM

Updated Data with 2023-01 counts is ready. First 2 tabs are most relevant.

The counts for English edits are very low, checked event_comment for en values and counts match, need to investigate.

SNowick_WMF moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Feb 7 2023, 6:04 PM
Sharvaniharan subscribed.Feb 8 2023, 10:21 PM

Hi @SNowick_WMF

As discussed, the timestamps attached to these two events can give you a good time_spent value:

Start event: [User clicking on the article descriptions task in SE:

{
  "$schema": "/analytics/mobile_apps/android_breadcrumbs_event/1.0.0",
  "meta": {
    "stream": "android.breadcrumbs_event",
    "id": "8fe7137e-0e11-4375-9e3a-61ff9a3ae1cc",
    "dt": "2023-02-08T21:54:59.705Z",
    "request_id": "3b9b8070-a7fb-11ed-be74-832ea79fb2f0",
    "topic": "eqiad.android.breadcrumbs_event",
    "partition": 0,
    "offset": 47987
  },
  "dt": "2023-02-08T21:54:59.631Z",
  "is_anon": false,
  "app_session_id": "d1f7f7f2514f8c82905e",
  "app_install_id": "19897815-0282-4f40-9564-7424acc4a745",
  "screen_name": "SuggestionsActivity.SuggestedEditsCardsFragment",
  "action": "back",
  "app_primary_language_code": "en",
  "http": {
    "request_headers": {
      "user-agent": "WikipediaApp/2.7.50428-dev-2023-02-08 (Android 12; Phone; Pixel 3 Build/SP1A.210812.016.C1) Developer Channel"
    },
    "client_ip": "98.51.231.41"
  }
}

and end event of clicking back from SE task:

{
  "$schema": "/analytics/mobile_apps/android_breadcrumbs_event/1.0.0",
  "meta": {
    "stream": "android.breadcrumbs_event",
    "id": "8fe7137e-0e11-4375-9e3a-61ff9a3ae1cc",
    "dt": "2023-02-08T21:54:59.705Z",
    "request_id": "3b9b8070-a7fb-11ed-be74-832ea79fb2f0",
    "topic": "eqiad.android.breadcrumbs_event",
    "partition": 0,
    "offset": 47987
  },
  "dt": "2023-02-08T21:54:59.631Z",
  "is_anon": false,
  "app_session_id": "d1f7f7f2514f8c82905e",
  "app_install_id": "19897815-0282-4f40-9564-7424acc4a745",
  "screen_name": "SuggestionsActivity.SuggestedEditsCardsFragment",
  "action": "back",
  "app_primary_language_code": "en",
  "http": {
    "request_headers": {
      "user-agent": "WikipediaApp/2.7.50428-dev-2023-02-08 (Android 12; Phone; Pixel 3 Build/SP1A.210812.016.C1) Developer Channel"
    },
    "client_ip": "98.51.231.41"
  }
}
JTannerWMF updated the task description. (Show Details)Mar 24 2023, 2:45 PM
Sharvaniharan added a comment.Mar 29 2023, 6:46 PM

The above events have been revised.. switched to .start and .end

Sharvaniharan added a comment.EditedMar 31 2023, 6:37 AM

@SNowick_WMF The document is now up to date with all events changed according how we had discussed this morning:
https://docs.google.com/spreadsheets/d/1mN_xXcn9MOAs4oYNFmKRnMG1BUYdhIZWdNhx0mPwooY/edit?usp=sharing

SNowick_WMF updated the task description. (Show Details)Mar 31 2023, 10:35 PM

As discussed with Jazmin the minimum requirements for the AB test are as follows:
We reach 4000 descriptions complete by at least 2000 unique users Deck

JTannerWMF moved this task from Needs Analytics to Ready for release on the Wikipedia-Android-App-Backlog (Android Release - FY2023-24) board.Jul 11 2023, 11:11 PM
Dbrant closed this task as Resolved.Jul 12 2023, 12:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL