Page MenuHomePhabricator

Add an image: experiment
Closed, ResolvedPublic

Description

Corresponding task from "add a link": T277356: Add a link: experiment


When we first deploy "add an image", we want to deploy it with a controlled experiment, somewhat similarly to "add a link". But the experiment will only happen amongst mobile users. Here are the details:

  • With the advent of the images experiment, we want to end the link experiment.
    • Ending the link experiment means making it so that all legacy accounts with Growth features and all future accounts with Growth features have this situation:
      • Replace the unstructured link task with the structured one. It will no longer be possible to do the unstructured one, though it can remain configurable in community configuration, in case a wiki wants to switch back.
      • If a wiki never had the unstructured link task because they had no applicable templates, then they should still get the structured one added.
      • Legacy users difficulty filter settings should not be changed, but new accounts should get the easy tasks by default ("copyedit" and "add a link").
    • We could either choose a date to end the link experiment across all wikis, before we start the image experiment. Or we could end the link experiment as part of turning on the images experiment wiki by wiki. This can be up to engineers.
  • For the image experiment, these are the groups that should be applied for account creations after the experiment starts:
    • Mobile A: these users created their account on mobile. They get "add an image" on as the only default task. (40% of mobile account creations). This is the only group that will have access to "add an image" at all. Some of these users may subsequently visit their homepage on desktop. See this task about how we will prevent them from using "add an image" on desktop: T295000: Add an image: prevent desktop usage
    • Mobile B: these users created their account on mobile. They have the default suggested edits experience without "add an image" in their difficulty filters at all. Their experience would include copyedit and "add a link" as their default tasks. (40% of mobile account creations)
    • Mobile C: these users created their account on mobile. They don't get the Growth features at all and are our usual control group. (20% of mobile account creations)
    • Desktop B: these users created their account on desktop. They have the default suggested edits experience without "add an image" in their difficulty filters at all. Their experience would include copyedit and "add a link" as their default tasks. (80% of desktop account creations)
    • Desktop C: these users created their account on desktop. They don't get the Growth features at all and are our usual control group. (20% of desktop account creations)
  • For accounts created before the experiment starts, "add an image" should not be added as a new task type. This is actually different than "add a link", where we did add it for legacy accounts.

Event Timeline

@MMiller_WMF could you please fill in the description for this task, using T277356 as a reference?

Some things that would be good to clarify:

  • What happens to the 'linkrecommendation' experiment group? Do we still need to keep that, or can link recommendation graduate out of the experimental mode?
  • unlike Add-Link, there's no unstructured task to compare against. So would we assign 100% of homepage users to the image-recommendation variant? Or would we want to compare some percentage of homepage users with add image as their default against homepage users without add image?
MMiller_WMF renamed this task from Add an image: Experiment to Add an image: experiment.Nov 2 2021, 11:39 PM
MMiller_WMF updated the task description. (Show Details)

@kostajh -- thank you for making this task. I talked with @nettrom_WMF and filled in the task description. How does it look? Would you prefer if I create a separate task for turning off the "add a link" experiment.

kostajh triaged this task as High priority.Nov 4 2021, 1:04 PM

If a wiki never had the unstructured link task because they had no applicable templates, then they should still get the structured one added.

That means we'll have to train the model and deploy for lots of wikis, but I thought further rollout of Add-Link was blocked per the latest comments in T290011: Deploy Add a link to a third round of wikis?

If a wiki never had the unstructured link task because they had no applicable templates, then they should still get the structured one added.

That means we'll have to train the model and deploy for lots of wikis, but I thought further rollout of Add-Link was blocked per the latest comments in T290011: Deploy Add a link to a third round of wikis?

From meeting discussion just now – there is not rollout of link-recommendation to more wikis; just promotion of link-recommendation out of experimental phase for wikis it is already deployed to.

Change 737680 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] End linkrecommendation variant experiment

https://gerrit.wikimedia.org/r/737680

Change 737714 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] [WIP] Allow for platform specific variant assignment

https://gerrit.wikimedia.org/r/737714

Change 737680 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] End linkrecommendation variant experiment

https://gerrit.wikimedia.org/r/737680

Change 738998 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] labs: Setup GEHomepageNewAccountVariantsByPlatform

https://gerrit.wikimedia.org/r/738998

Change 738999 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Set up GEHomepageNewAccountVariantsByPlatform

https://gerrit.wikimedia.org/r/738999

Change 739000 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Start imagerecommendation variant experiment

https://gerrit.wikimedia.org/r/739000

Change 738998 merged by jenkins-bot:

[operations/mediawiki-config@master] labs: Setup GEHomepageNewAccountVariantsByPlatform

https://gerrit.wikimedia.org/r/738998

Change 738999 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Set up GEHomepageNewAccountVariantsByPlatform

https://gerrit.wikimedia.org/r/738999

Mentioned in SAL (#wikimedia-operations) [2021-11-16T00:19:40Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 50d9f2687cd11e6f838313a530c6bbd498d0b83e: GrowthExperiments: Set up GEHomepageNewAccountVariantsByPlatform (T294737) (duration: 00m 56s)

Change 737714 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Allow for platform specific variant assignment

https://gerrit.wikimedia.org/r/737714

Change 739032 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] GrowthExperiments configuration fixes

https://gerrit.wikimedia.org/r/739032

Change 739000 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Start imagerecommendation variant experiment

https://gerrit.wikimedia.org/r/739000

Change 742454 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Revert \"GrowthExperiments: Start imagerecommendation variant experiment\"

https://gerrit.wikimedia.org/r/742454

Change 742454 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert \"GrowthExperiments: Start imagerecommendation variant experiment\"

https://gerrit.wikimedia.org/r/742454

Change 739032 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments configuration fixes

https://gerrit.wikimedia.org/r/739032

Mentioned in SAL (#wikimedia-operations) [2021-12-02T19:12:20Z] <taavi@deploy1002> Synchronized wmf-config: Config: [[gerrit:739032|GrowthExperiments configuration fixes (T294737)]] (duration: 00m 57s)