Page MenuHomePhabricator

Add a link: quality gates
Open, MediumPublic

Description

NOTE: this is not slated for the initial release

To deter careless editing and vandalism through "add a link", we may want to include some "quality gates" to identify if a user appears to be editing carelessly, and then prompt them to change or slow their behavior. This task would need more design and discussion before becoming actionable. We will likely want to evaluate data from the initial release before taking action.

Possible triggers

  • User has accepted all suggestions on three articles in a row.
  • User accepts over 90% of suggestions (given that we believe the model is about 80% accurate).
  • User is spending less than four seconds per suggestion on average.
  • User has over 10 "add a link" edits with a revert rate of 30% or more.

Possible interventions

  • Dialog that says "You've been accepting all the AI suggestions you've received. Make sure that the AI isn't making mistakes."
  • Dialog that says "You're reviewing links very quickly! Slow down to make sure the AI isn't making mistakes."
  • Pausing/Not showing Add links tasks for a period of time to that person (with some message letting them know why and how to improve once "timeout" is over)
  • Provide a short user education tutorial they must complete before being able to review add links tasks.

Event Timeline

kostajh triaged this task as Medium priority.May 27 2021, 11:54 AM
kostajh moved this task from Backlog to Post-release backlog on the Add-Link board.

Change 737402 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Implement dailyLimit quality gate for add link

https://gerrit.wikimedia.org/r/737402

Change 737489 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[schemas/event/secondary@master] LinkSuggestionInteraction: Add qualitygate_dialog

https://gerrit.wikimedia.org/r/737489

@RHo @MMiller_WMF this doesn't need urgent attention, but if you want, the same daily limit quality gate from image recommendation could be added for link recommendation tasks. Like the image recommendation quality gate, it will be configurable via Special:EditGrowthConfig once T294518: Add Image: Allow disabling feature via community configuration is done -- both to enable/disable the quality gate, as well as to specify the maximum number of tasks per day per user.

Functionally, it's the exact same behavior as T293316: Add an image: minimal quality gate except applied to link-recommendation tasks. The only differences are:

@kostajh -- we're so glad you did this. It's one of the top requests about "add a link", so if we're able to just implement, let's do it! We would want to implement with the same 25 task limit that we used for images. 25 is a number that communities proposed to us for this, so we want to start there.

And the copy as you altered it looks good to us.

My only question is whether the two quality gates will work well with each other. Like if someone completes all their image tasks, then switches over to links and completes all their link tasks, will both the gates function fine together and stop the user for both task types as they navigate their feed?

If all good, then please go ahead and deploy!

@kostajh -- we're so glad you did this. It's one of the top requests about "add a link", so if we're able to just implement, let's do it! We would want to implement with the same 25 task limit that we used for images. 25 is a number that communities proposed to us for this, so we want to start there.

And the copy as you altered it looks good to us.

sounds good! Updated the patch to use the 25 limit.

My only question is whether the two quality gates will work well with each other. Like if someone completes all their image tasks, then switches over to links and completes all their link tasks, will both the gates function fine together and stop the user for both task types as they navigate their feed?

If all good, then please go ahead and deploy!

Yeah, no problems there. If the user does 25 for each task type, they'll be stopped from both until the next day.

Change 737489 merged by jenkins-bot:

[schemas/event/secondary@master] LinkSuggestionInteraction: Add qualitygate_dialog

https://gerrit.wikimedia.org/r/737489

Change 737402 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Implement dailyLimit quality gate for add link

https://gerrit.wikimedia.org/r/737402