Page MenuHomePhabricator

Section-level images: Instrumentation
Open, In Progress, MediumPublic

Description

User story:

As the Growth team, I want to be able to measure the task funnel and effectiveness of this newcomer task, because then I can make decisions about improving the task further or scaling the feature to more wikis.


For section-level "add an image" instrumentation, we want to follow the same instrumentation as is used for the article-level "add an image" task.

T294659: Add an image: measurement


Acceptance criteria:

Given I'm completing a section-level "add an image" task,
When I complete any action within this structured task,
Then the following data is captured:

  • User ID
  • Distinguish by wiki.
  • Analyze how much time elapses between each action.
  • Know whether each action was taken from desktop or mobile.
  • Know which article the user is on (through page title and page id).
  • Include timestamps with millisecond precision.

Event Timeline

@nettrom_WMF would you like separate schemas (where it makes sense, e.g. mediawiki/page/image-suggestions-feedback) or should we just plug in a section ID into the schema, and set it to null for the old image recommendation task type? Do you want a specific format for the section ID (ordinal, section title text etc)?

@nettrom_WMF would you like separate schemas (where it makes sense, e.g. mediawiki/page/image-suggestions-feedback) or should we just plug in a section ID into the schema, and set it to null for the old image recommendation task type? Do you want a specific format for the section ID (ordinal, section title text etc)?

I think we should reuse as much of our schemas and instrumentation as we can since the workflow is almost identical to that of the old Add an Image task. While perhaps a bit redundant, capturing both the ordinal and the section title would give us a much information as possible to work with in case we need to dig deeper (e.g. the text makes it easier to find the section, the ordinal could give us information about whether earlier sections are more likely to have/get images). Then those can be set to null for the old task, as you suggest.

Change 924879 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[schemas/event/secondary@master] Add schema for section_image_suggestion_interaction

https://gerrit.wikimedia.org/r/924879

Change 924881 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] Section images: use new interaction logger

https://gerrit.wikimedia.org/r/924881

Change 926468 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[schemas/event/secondary@master] Add section title and ordinal in image suggestions submission events

https://gerrit.wikimedia.org/r/926468

Sgs changed the task status from Open to In Progress.Jun 2 2023, 1:19 PM
Sgs moved this task from In Progress to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.

Change 924879 merged by jenkins-bot:

[schemas/event/secondary@master] Add section title and ordinal in the image_suggestion_interaction schema

https://gerrit.wikimedia.org/r/924879

Change 924881 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Section images: bump instrumentation schema to 1.4.1

https://gerrit.wikimedia.org/r/924881

Change 926468 merged by jenkins-bot:

[schemas/event/secondary@master] Add section title and ordinal in image suggestions submission events

https://gerrit.wikimedia.org/r/926468

Change 928800 had a related patch set uploaded (by Elukey; author: Elukey):

[eventgate-wikimedia@master] Bump secondary schema repository commit sha

https://gerrit.wikimedia.org/r/928800

Change 928800 merged by Milimetric:

[eventgate-wikimedia@master] Bump secondary schema repository commit sha

https://gerrit.wikimedia.org/r/928800

Change 929164 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Deploy the new eventgate-wikimedia container image

https://gerrit.wikimedia.org/r/929164

Change 929164 merged by jenkins-bot:

[operations/deployment-charts@master] Deploy the new eventgate-wikimedia container image

https://gerrit.wikimedia.org/r/929164

@KStoller-WMF, @Etonkovidova is out at on offsite this week. Would you like to assign QA of this one to an engineer, or do you want to review the events emitted via interactions with add image and add section image to make sure everything's as you'd expect?

@nettrom_WMF - at our 1:1 today, let's QA this together, so you can ensure we are gathering the data you need and I can learn how to review instrumentation events.

I introduced console-based event QA to @KStoller-WMF in our 1:1. We went through most of the workflow and checked the various events. From what we found the instrumentation is working correctly, there's nothing there that blocks deployment.

I believe I found a discrepancy between the documentation and the event data. https://gerrit.wikimedia.org/r/924879 says that section data is included for all section-level events, but from what I found it's only the action = "impression" event with active_interface = "machinesuggestions_mode" that contains section-level data in action_data. From my point of view, I think it's fine to keep the events the way they are but update the schema documentation to reflect that it's the impression event that defines the session as either "section level" – meaning action_data contains the section ordinal and title – or "article level" meaning there's no additional data.