Page MenuHomePhabricator

Count edits/users for “Suggested edits” explore feed and article/gallery CTA
Closed, ResolvedPublic

Assigned To
Authored By
scblr
Dec 3 2019, 12:27 PM
Referenced Files
F31455137: eaf-27.png
Dec 3 2019, 12:27 PM
F31455148: eaf-ic-13.png
Dec 3 2019, 12:27 PM
F31455141: eaf-ic-38.png
Dec 3 2019, 12:27 PM
F31455139: eaf-ic-35.png
Dec 3 2019, 12:27 PM

Description

The Android team have experimented with a number of ways to alert users about the Suggested edits feature. We would like to assess the relative effectiveness of these methods so that we can prioritise our work for Q3. Ideally we would like to have at least directional answers before the holiday break.

01)

Product decision: Should we get rid of Explore feed cards in Suggested edits V4?

eaf-27.png (1×720 px, 675 KB)
eaf-ic-35.png (1×720 px, 868 KB)

Data request:

  • Count edits and users who edited via Suggested edits Explore feed card in total and per edit type (image captions / article descriptions)
  • Put these numbers into perspective with total app edits, Suggested edits, users that edit with the app and Suggested edits users

Data report:
https://analytics.wikimedia.org/published/reports/wikipedia-android-app//T239706_Report_Part_1.html


02)

Product decision: Should we change the current concept of teasing image related edits in V4? (currently image captions)

eaf-ic-38.png (2×1 px, 2 MB)
eaf-ic-13.png (1×824 px, 511 KB)

Data request:

  • Count edits and users who edited via Suggested edits “Add image caption” CTA on article page
  • Count edits and users who edited via Suggested edits “Add image caption” CTA in gallery module
  • Put these numbers into perspective with total app edits, Suggested edits, users that edit with the app and Suggested edit users

Related
T239690

Done
  • Explore feed card CTR and conversion (Part 1)
  • Image caption promotion CTR and conversion (Part 2) (data was missing, need to verify that bug is fixed and that we have data to measure/compare)

Event Timeline

As discussed last week @Charlotte, here are the data requests! 👋

Hey @kzimmerman - is this something we can get on someone's to-do list before the holidays?

LGoto triaged this task as High priority.Dec 9 2019, 5:37 PM
LGoto removed a project: Epic.

@Charlotte we're not going to be able to get these numbers before the holidays.

Just in case there's something already available, @mpopov is going to take a look and see if we have existing data that speaks to your questions. (Although the answer may be no.)

Shay is starting next Monday and this could be one of her first onboarding projects, with support from team members. Timing-wise, that means you'll start to see some numbers in mid-January (allowing for time to onboard, assuming that she gets database access fairly quickly.)

Thanks for letting me know @kzimmerman. The sooner the better obvs.

Count edits and users who edited via Suggested edits “Add image caption” CTA on article page
Count edits and users who edited via Suggested edits “Add image caption” CTA in gallery module

There is currently no way to determine this because the only related events that trigger on the article page and the gallery module are MobileWikiAppEdit events which don't include any information about how the user started making the wikidataDescriptionEdit. Recommendation: add source field to schema & funnel and put values like "article page", "gallery module", "suggested edits" in it.

@Dbrant @Charlotte: how feasible is it to release a version with such a change in it before the holidays so we could come back from the holidays to a couple of weeks of collected data to answer the second request?


Notes for future self or for whoever ends up working on this. For some reason cardClicked events aren't generated when clicking on FeedContentType.SUGGESTED_EDITS cards (but seemingly only those, clicks on other cards seem unaffected).

Here's an example of logged events starting with seeing the Suggested Edits card in the Explore Feed to tapping it, to canceling and going back to the feed:

2019-12-10 16:29:32.578 12150-13229/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():53:

{
  "schema": "MobileWikiAppFeed",
  "revision": 18115458,
  "wiki": "enwiki",
  "event": {
    "action": "cardShown",
    "cardType": 21,
    "language": "en",
    "time_spent": 14,
    "client_dt": "2019-12-10T16:29:32-0500",
    "app_install_id": "[REDACTED]",
    "session_token": "d1a24b2a-5082-45ce-b87a-4a94e80b5aa0"
  }
}

2019-12-10 16:30:06.777 12150-13229/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():53:

{
  "schema": "MobileWikiAppFeed",
  "revision": 18115458,
  "wiki": "enwiki",
  "event": {
    "action": "exit",
    "time_spent": 48,
    "client_dt": "2019-12-10T16:30:06-0500",
    "app_install_id": "[REDACTED]",
    "session_token": "d1a24b2a-5082-45ce-b87a-4a94e80b5aa0"
  }
}

2019-12-10 16:30:06.830 12150-13228/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():53:

{
  "schema": "MobileWikiAppEdit",
  "revision": 18115551,
  "wiki": "enwiki",
  "event": {
    "action": "start",
    "wikidataDescriptionEdit": "new",
    "anon": false,
    "client_dt": "2019-12-10T16:30:06-0500",
    "app_install_id": "[REDACTED]",
    "session_token": "b5e0340b-d2ed-4d9d-af28-dff26847c952"
  }
}

2019-12-10 16:30:06.988 12150-13207/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():53:

{
  "schema": "MobileWikiAppEdit",
  "revision": 18115551,
  "wiki": "enwiki",
  "event": {
    "action": "ready",
    "wikidataDescriptionEdit": "new",
    "anon": false,
    "client_dt": "2019-12-10T16:30:06-0500",
    "app_install_id": "[REDACTED]",
    "session_token": "b5e0340b-d2ed-4d9d-af28-dff26847c952"
  }
}

2019-12-10 16:30:43.305 12150-13229/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():53:

{
  "schema": "MobileWikiAppSuggestedEdits",
  "revision": 18949003,
  "wiki": "enwiki",
  "event": {
    "edit_tasks": {
      "add-caption": {
        "cancels": 0,
        "clicks": 0,
        "failures": 0,
        "impressions": 0,
        "successes": 0,
        "suggestions_clicked": 0
      },
      "add-description": {
        "cancels": 1,
        "clicks": 1,
        "failures": 0,
        "impressions": 1,
        "successes": 0,
        "suggestions_clicked": 1
      },
      "translate-caption": {
        "cancels": 0,
        "clicks": 0,
        "failures": 0,
        "impressions": 0,
        "successes": 0,
        "suggestions_clicked": 0
      },
      "translate-description": {
        "cancels": 0,
        "clicks": 0,
        "failures": 0,
        "impressions": 0,
        "successes": 0,
        "suggestions_clicked": 0
      }
    },
    "help_opened": 0,
    "scorecard_opened": 0,
    "source": "feed",
    "time_spent": 70,
    "client_dt": "2019-12-10T16:30:43-0500",
    "app_install_id": "[REDACTED]",
    "session_token": "3e9c576c-1411-477b-ada4-155a67a49938"
  }
}

@kzimmerman: MobileWikiAppSuggestedEdits table will have events where source = 'feed' (like the event seen directly above) and that can be used to count how many edits were made & how many users edited from the Suggested Edits card in the Explore Feed.

Thanks for investigating, @mpopov!

@Charlotte - Mikhail will work with Shay on this as part of Shay's onboarding. I'm hoping Shay will have systems access by the start of January.

Thanks @mpopov

Count edits and users who edited via Suggested edits Explore feed card in total and per edit type (image captions / article descriptions)
Count edits and users who edited via Explore feed card in total and per edit type (image captions (article descriptions)

@Charlotte @schoenbaechler: what's the difference between those two asks?

There was none, thanks @Charlotte for removing the redundancy in the task’s description.

Recommendation: add source field to schema & funnel and put values like "article page", "gallery module", "suggested edits" in it.

Sounds good, will mention it at standup and ask about a possible timeline.

For some reason cardClicked events aren't generated when clicking on FeedContentType.SUGGESTED_EDITS cards (but seemingly only those, clicks on other cards seem unaffected).

Will be fixed in T240469

SNowick_WMF updated the task description. (Show Details)
SNowick_WMF updated the task description. (Show Details)

Report for part 1 here:
https://analytics.wikimedia.org/published/reports/wikipedia-android-app/T239706_Report_Part_1.html

Percent of all Edits by Source: (14 Day sample)
14.84% Feed
85.16% Menu/Suggested

When data is available will report on part 2.

SNowick_WMF updated the task description. (Show Details)
SNowick_WMF updated the task description. (Show Details)

Hi @schoenbaechler I was able to pull some data for the 2nd request but I have some questions because data isn't showing exactly what we would expect and I want to troubleshoot.

For “Add image caption” CTA on article page and “Add image caption” CTA in gallery module - the user would only be able to edit an image caption in that funnel, correct? We are seeing actions originating from gallery (but not article, which I also need to investigate) but they don't look like image edits.

Is this CTA still live?

Report for part 1 here:
https://analytics.wikimedia.org/published/datasets/wikipedia-android-app-reports/T239706_Report_Part_1.html

Hi @SNowick_WMF, thanks! Going to study part 1 of the report today.

For “Add image caption” CTA on article page and “Add image caption” CTA in gallery module - the user would only be able to edit an image caption in that funnel, correct?

Yes, that’s correct.

Is this CTA still live?

Yes, CTA’s in the article header image and in gallery view are live.

I’m positively surprised by the 14.84% that come from the Explore feed. I think we should continue to create cards for the feed, as people are using it and it doesn’t hurt. Do you agree @Charlotte? Will create designs for it today/tomorrow in T244866.

Thanks @SNowick_WMF!

But it seems like edits coming from the Feed have the lowest conversion rate, and the lowest absolute count. Shouldn't we focus on features/interfaces with more potential throughput?

people are using it

Are we certain that 167 total edits in a two-week period merits the proliferation of more cards? And do we know whether all of these edits are from unique users, instead of a single person making 100 edits? (cc @SNowick_WMF )

it doesn’t hurt

Let's keep in mind that it hurts in terms of engineering capacity, since the cards take a nontrivial amount of effort to build, and then take a nontrivial effort to maintain, which keeps compounding with other features we add.

SNowick_WMF closed this task as Resolved.EditedFeb 13 2020, 4:39 AM

Report is here: https://analytics.wikimedia.org/published/reports/wikipedia-android-app/T239706_Report_Part_2.html

Key takeaways:
From gallery the average daily 'start' event was 230 and daily completed was 72, with an average completion rate of 30.1%.

From article the average daily 'start' event was 706 and daily completed was 81, with an average completion rate of 11.3%.

For comparison, image edits from SuggestedEdits average daily 'start' event was 481 and daily completed was 130, with an average completion rate of 26.5%.