Page MenuHomePhabricator

Add an image: quality gates
Open, Needs TriagePublic

Description

In T274325: Add a link: quality gates, we discussed some ideas for quality gates for "add a link", which would be rules that slowed or stopped contributors who are adding many damaging edits to the wiki. We have not needed to implement any of them yet for "add a link".

But adding images to articles is a highly impactful and visible edit. That means that a user who is doing a bad job could cause a lot of damage quickly through "add an image". We've seen this through the Wikipedia Pages Wanting Photos campaign, in which most users were being productive, but some generated many weak edits. Discussions at English Wikipedia here led that community to limit participants to 25 of these edits per day -- a blunt way to limit a user who is moving quickly.

For "add an image", we've had ideas that range from simple to sophisticated. With each of these ideas, we would either stop the users from doing the task, try to re-onboard them, or suggest different tasks.

  • Quantity: limit participants to a certain number of edits per day.
  • Reverts: limit participants who have had too many reverts to this task type.
  • Speed: limit participants based on the average time they're spending on each edit being too low.
  • Ratio: limit participants based on them answering "yes" or "no" more often than is expected from the algorithm.

In Iteration 1, we'll do the simple "quantity" gate, with this rule:

Users may not open a 26th suggestion after having submitted "yes" or "no" responses on 25 suggestions in that day.

  • Though both "yes" and "no" count, "skip" does not.
  • The "day" period should be defined as midnight-to-midnight in the user's timezone. After midnight, it resets.

In terms of user experience, we need to decide...

  • ...what the user sees in the post-edit dialog after they complete their 25th suggestion that day (T290789: Add an image: post-edit dialog).
  • ...what happens in the suggested edits feed if the user is viewing it and the difficulty filters after their 25th suggestion that day.
  • ...what happens when the user has multiple tasks opens in multiple tabs, and tries to save a task after reaching their quota in an another tab.
  • ...what happens when the user has a tasks open in a tabs, and the homepage open in another, and tries to select an image task after reaching their quota in an another tab.

Other points:

  • If trivial, we would like to have the option to apply this same logic to "add a link" and the other unstructured task types.
  • Parameters around this quality gate should become part of community configuration.
    • For which task types the gate is active.
    • What threshold it should use for number of edits.

Event Timeline

Assigned to @RHo to think about these parts:

  • We need to decide what the user sees in the post-edit dialog after they complete their 25th suggestion that day (T290789: Add an image: post-edit dialog).
  • We need to decide what happens in the suggested edits feed if the user is viewing it and the difficulty filters after their 25th suggestion that day.

We'll also need UX for users visiting a task when having reached the quota, or trying to save a task when having reached the quota.

@Tgr -- I was thinking that once the user reached the quota, we would not offer them any way to open new image tasks (e.g. they might be grayed out and unclickable in the feed), and so we would not need UX for visiting when having reached the quota. Same for trying to save a task -- we wouldn't let them get that far. But perhaps @RHo will prefer a UX where they are allowed to get that far.

Or are you saying we need to be able to handle those cases if, say, the user has opened multiple tabs of tasks and meets the quota with one tab, and then tries to save in another tab?

Exactly, they might e.g. have the homepage open and do tasks in new tabs. Updating the state of one open tab in real time based on a change in another tab is not something MediaWiki currently provides any support for, and doing it from scratch seems way more effort than it's worth.

Hi there - here are the proposals I have for the 'quantity gate' behaviour in the following scenarios (it's also on the Figma file here and here):

ScreenScenarioInteraction details + Mock as of 2021-10-05
Post-edit dialog1. When a user has added their final Add image edit quota for the dayUser is shown another task type and also can return to SE module with Add images task removed from the queue
image.png (1×684 px, 220 KB)
updated image to show both 'no change' and image added' dialogs
Add image task flow2. User who has reached quota has another add image task open in another tab. User makes a selection on the image inspectorAt this and every other point during the add image flow (progress from Caption to preview summary, or from summary to publish), there is a check on whether the quota is reached an this error message is shown.
image.png (1×720 px, 243 KB)
Newcomer homepage3. User who has reached quota and has Newcomer homepage open on another tab with an “Add an image” task shown. User opens SE module.Message is shown on the SE module that lets the user know that the preview article is no longer available, which on close refreshes the page to no longer show Add image tasks.
image.png (1×1 px, 202 KB)
Suggested edits module4. User who has reached quota and has Suggested edits open on another tab. User selects an Add an image task.Same message as above, which lets the user know that the preview article is no longer available, which on close refreshes the page to no longer show Add image tasks.
image.png (1×720 px, 115 KB)

Part of my proposal is that we continue to show the "Add image task" checked in the task difficulty filter, but it would merely not show any more as being available for the rest of the day.

Thank you, @RHo. I think these designs will work well. I have three questions for you and @Tgr.

No more tasks in the feed

@RHo, you wrote, "we continue to show the "Add image task" checked in the task difficulty filter, but it would merely not show any more as being available for the rest of the day". Do you mean that no more of the tasks even appear in the feed? Or that they're there, but when the user taps them, they get the dialog saying they've reached their quota? The former may involve engineering challenges around caching.

When no other task types are selected

An important underlying open question is: what to do when the user finishes their image tasks, and that's the only task type they have selected (or they otherwise have no other tasks in their feed)? Some options are:

  • Automatically change their task selections so that some other task types are in their feed, e.g. re-select all the "Easy" tasks. This may be annoying/confusing to a user who selects just image tasks, completes them all every day, and then intends to come back the next day to do more image tasks. "Why did my tasks change and I have to change them back every day?" I think this will be complicated to administer, especially since wikis have inconsistent task types.
  • Don't do anything to their task selections, but just let them encounter a "no tasks available" message in the suggested edits feed, which could be tailored to "you have completed all image suggestions", and prompt them to change their task types themselves. But what would this mean for Scenario #1, in the post-edit dialog? Should we use the task card space to tell them to go to the homepage to switch their task types? With classic tasks, if the user only had one task in their feed and they complete it, the post-edit dialog offers no task and looks like this:

image.png (606×1 px, 293 KB)

Scenario #3

Another question I have is about Scenario #3 ("User who has reached quota and has Newcomer homepage open on another tab with an “Add an image” task shown. User opens SE module.") It sounds like for this scenario, you're saying that the first time the user opens the module after meeting their quota, they see this dialog and then dismiss it (so that they are informed about why no more image tasks are in their feed). I think it would be good to have an experience that doesn't require us to show something just once. I propose that we don't have special handling for this scenario. The user will already have been informed that they hit their quota through the post-edit dialog, and we could just handle the same way as Scenario #4.

There are four points it's both logical and easy to "gate" the user, I think:

  • When the user opens the homepage: once the user is past the quota, we filter out Add Image tasks. Possibly alongside other messaging (such as a customized "no results" message or some change to the task selector). We can't easily update the homapage once it's already open; we should just rely on Scenario #2 for that. We also can't easily differentiate between "user reached the quota and would have more Add Image tasks otherwise" and "user reached the quota but there aren't more Add Image tasks anyway".
    • Also, when the homepage is already open but the user reloads the tasks (by changing filter settings), we can filter out Add Image tasks and potentially show a custom error message but I'd rather avoid modifying parts of the interface outside the task queue at this point.
  • When the user opens an Add Image task. We can easily reuse the "no suggestion found" dialog with a slightly different message here.
  • When the user tries to save an Add Image task. Again, we can reuse the existing error dialog here.
  • In the post-edit dialog.

Some questions that come to mind:

  • What happens if the user reaches the quota with a "no" answer? IIRC we don't plan to show the post-edit dialog in that case.
  • What happens if the user is past the quota but has selected multiple task types so the queue is not empty? Should we try to explain somehow why there are no Add Image tasks in the queue?
  • The "day" period should be defined as midnight-to-midnight in the user's timezone. - how should this work when the user can make different edits from different time zones? Also, how do we tell the user's time zone? GeoIP? The time zone in their user preferences (which I think defaults to the timezone of the content language, or UTC for more dispersed languages)?

Do you mean that no more of the tasks even appear in the feed? Or that they're there, but when the user taps them, they get the dialog saying they've reached their quota? The former may involve engineering challenges around caching.

Thanks for being mindful of caching! :) (One of the hard things in computer science, as the saying goes.) In this case it's not too bad, we'd just have to clear the cache after the user accepts/rejects a task, something we might want to do anyways.

Another approach would be not filtering out the Add Image tasks but showing them in some alternative way (e.g. with a lock icon), though maybe that's a bit in-your-face?

The "day" period should be defined as midnight-to-midnight in the user's timezone. - how should this work when the user can make different edits from different time zones? Also, how do we tell the user's time zone? GeoIP? The time zone in their user preferences (which I think defaults to the timezone of the content language, or UTC for more dispersed languages)?

It would definitely be less complicated to say "Wait 12 hours" (the "12 hours" string can be updated so when they come back an hour later, it says "11 hours" or whatever), if that is acceptable from a product perspective.

A meta point is that given the various permutations and scenarios and our desire to release something in early November, we might think about the simplest version of this for initial release, then build out a more comprehensive UX around this in further iterations.

IMO the simplest version would only handle the case where the user clicks on a card from Special:Homepage; we'd show them a dialog directly on Special:Homepage saying that they need to wait X hours until doing more Add Image tasks and suggest that they try other tasks. That would still allow a user to get around the gate if they already had another tab open with an image suggestion task, but it does not sound especially problematic to allow one extra edit to go through.

Hiya, I've tried to group and respond to different comments below:

A. No more tasks in the feed

No more tasks in the feed
@RHo, you wrote, "we continue to show the "Add image task" checked in the task difficulty filter, but it would merely not show any more as being available for the rest of the day". Do you mean that no more of the tasks even appear in the feed? Or that they're there, but when the user taps them, they get the dialog saying they've reached their quota? The former may involve engineering challenges around caching.

Thanks for being mindful of caching! :) (One of the hard things in computer science, as the saying goes.) In this case it's not too bad, we'd just have to clear the cache after the user accepts/rejects a task, something we might want to do anyways.
Another approach would be not filtering out the Add Image tasks but showing them in some alternative way (e.g. with a lock icon), though maybe that's a bit in-your-face?

My meaning was indeed that the tasks do not appear in the feed, because those task types are not there anymore (similar to having a topic interest selected but not having any more selected task types for "food and drink" that day). The reason is that it seems both a less good experience to continue to show tasks that are not actually actionable (either through showing a error message on click, or a locked icon), and a wasted opportunity in not providing a different task suggestion to do.

B. When no other task types are selected

An important underlying open question is: what to do when the user finishes their image tasks, and that's the only task type they have selected (or they otherwise have no other tasks in their feed)? Some options are:

  • Automatically change their task selections so that some other task types are in their feed, e.g. re-select all the "Easy" tasks. This may be annoying/confusing to a user who selects just image tasks, completes them all every day, and then intends to come back the next day to do more image tasks. "Why did my tasks change and I have to change them back every day?" I think this will be complicated to administer, especially since wikis have inconsistent task types.
  • Don't do anything to their task selections, but just let them encounter a "no tasks available" message in the suggested edits feed, which could be tailored to "you have completed all image suggestions", and prompt them to change their task types themselves. But what would this mean for Scenario #1, in the post-edit dialog? Should we use the task card space to tell them to go to the homepage to switch their task types? With classic tasks, if the user only had one task in their feed and they complete it, the post-edit dialog offers no task and looks like this:

image.png (606×1 px, 293 KB)

My preference would be your first proposed option, that we enable another task type as an opportunity to encourage that person to level up/branch out to different edits. Perhaps we could use a simple rule that either another structured task is selected, or another medium task (depending on the wiki availability)? I don't think it will be annoying/confusing the next day for an avid add image user, because their quota will have reset, and the add images tasks would appear once again (in my proposal, we would continue to have the "Add images" task selected). If even when the Add links task / Other medium task turns up empty, that's when the existing behaviour kicks in, where there is no next suggestion in the post-edit dialog.

C. Scenario #3 ("User who has reached quota and has Newcomer homepage open on another tab with an “Add an image” task shown. User opens SE module.")

Another question I have is about Scenario #3 ("User who has reached quota and has Newcomer homepage open on another tab with an “Add an image” task shown. User opens SE module.") It sounds like for this scenario, you're saying that the first time the user opens the module after meeting their quota, they see this dialog and then dismiss it (so that they are informed about why no more image tasks are in their feed). I think it would be good to have an experience that doesn't require us to show something just once. I propose that we don't have special handling for this scenario. The user will already have been informed that they hit their quota through the post-edit dialog, and we could just handle the same way as Scenario #4.

That makes sense to me, I've updated on the Figma.

D. Newcomer homepage and/or Suggested edits module behaviour

There are four points it's both logical and easy to "gate" the user, I think:

  • When the user opens the homepage: once the user is past the quota, we filter out Add Image tasks. Possibly alongside other messaging (such as a customized "no results" message or some change to the task selector). We can't easily update the homapage once it's already open; we should just rely on Scenario #2 for that. We also can't easily differentiate between "user reached the quota and would have more Add Image tasks otherwise" and "user reached the quota but there aren't more Add Image tasks anyway".

Does this comment mean that the mock I have for scenario 3 and 4 would never happen, because there's no check at this point (on the Suggested edits module) whether the user has reached their quota or not yet? I was proposing that this check *would* be added so that the person finds out before they waste time and data on loading a task that is invalid.

  • Also, when the homepage is already open but the user reloads the tasks (by changing filter settings), we can filter out Add Image tasks and potentially show a custom error message but I'd rather avoid modifying parts of the interface outside the task queue at this point.

I think we are of the same opinion here but to confirm - do you mean that we would continue to allow the Add image task type to be selected in the filter settings, but would merely not should any "Add image" task results? I agree with this if so, I also think it is not necessary to show an extra message here as they should know from getting the message on the post-edit dialog that they've reached the max. for the day.

  • When the user opens an Add Image task. We can easily reuse the "no suggestion found" dialog with a slightly different message here.
  • When the user tries to save an Add Image task. Again, we can reuse the existing error dialog here.

Per my comment above, I would like to propose that we check whether the task is valid when on the Suggested edits task queue and show the message, rather than getting it after being taken to the task. It would also be an improvement to the Add links task if we add the check and error message on the Suggested edits page too (and obviously also be more consistent across tasks).

E. Post-edit dialog
  • In the post-edit dialog.

Some questions that come to mind:

  • What happens if the user reaches the quota with a "no" answer? IIRC we don't plan to show the post-edit dialog in that case.

We do plan to show a post-edit dialog when the image is either rejected or skipped. But you reminded me to add the quality gate version when the last quote with a no answer. T290789: Add an image: post-edit dialog

F. Explaining the quota on the Suggested edits feed (non-error scenario)
  • What happens if the user is past the quota but has selected multiple task types so the queue is not empty? Should we try to explain somehow why there are no Add Image tasks in the queue?

I don't think this is necessary since they will have seen the message advising them of the quote being reached before they got back to this point.

G. Timezone / Day period
  • The "day" period should be defined as midnight-to-midnight in the user's timezone. - how should this work when the user can make different edits from different time zones? Also, how do we tell the user's time zone? GeoIP? The time zone in their user preferences (which I think defaults to the timezone of the content language, or UTC for more dispersed languages)?

! In T290788#7404700, @kostajh wrote:

It would definitely be less complicated to say "Wait 12 hours" (the "12 hours" string can be updated so when they come back an hour later, it says "11 hours" or whatever), if that is acceptable from a product perspective.

I think it will be easier to understand to keep it as daily, without having to have any countdown clock logic. It also seems like an edge case we could let slip if people are so keen that they make extra edits by going to different timezones (even if someone in Australia changes their timezone back to Pacific time the previous day, they would conceivability only ever be over by 25 max. in a 2-day window). And keeping it simple, using the user preference timezone seems fine.

H. Cutting scope for November?

A meta point is that given the various permutations and scenarios and our desire to release something in early November, we might think about the simplest version of this for initial release, then build out a more comprehensive UX around this in further iterations.
IMO the simplest version would only handle the case where the user clicks on a card from Special:Homepage; we'd show them a dialog directly on Special:Homepage saying that they need to wait X hours until doing more Add Image tasks and suggest that they try other tasks. That would still allow a user to get around the gate if they already had another tab open with an image suggestion task, but it does not sound especially problematic to allow one extra edit to go through.

I think it's really important to show them the message on the post-edit dialog when the person hits the limit, so would definitely not want that cut out of the scope. And we already have an error message when the user reaches the Add image task that just needs copy tweaks to, so I feel like the additional message on the homepage means we would end up doing these 4 scenarios as the first version. Is that feasible?

D. Newcomer homepage and/or Suggested edits module behaviour

There are four points it's both logical and easy to "gate" the user, I think:

  • When the user opens the homepage: once the user is past the quota, we filter out Add Image tasks. Possibly alongside other messaging (such as a customized "no results" message or some change to the task selector). We can't easily update the homapage once it's already open; we should just rely on Scenario #2 for that. We also can't easily differentiate between "user reached the quota and would have more Add Image tasks otherwise" and "user reached the quota but there aren't more Add Image tasks anyway".

Does this comment mean that the mock I have for scenario 3 and 4 would never happen, because there's no check at this point (on the Suggested edits module) whether the user has reached their quota or not yet? I was proposing that this check *would* be added so that the person finds out before they waste time and data on loading a task that is invalid.

Yeah, we'd require some sort of cross-tab communication for that, and that's a performance and complexity cost I'd rather not incur for the initial iteration.

  • Also, when the homepage is already open but the user reloads the tasks (by changing filter settings), we can filter out Add Image tasks and potentially show a custom error message but I'd rather avoid modifying parts of the interface outside the task queue at this point.

I think we are of the same opinion here but to confirm - do you mean that we would continue to allow the Add image task type to be selected in the filter settings, but would merely not should any "Add image" task results? I agree with this if so, I also think it is not necessary to show an extra message here as they should know from getting the message on the post-edit dialog that they've reached the max. for the day.

I meant, if we want to strike out the task type in the task type filter or something like that, we could easily do it during the initial pageload, but not afterwards (if the state changes while the homepage is open).

  • When the user opens an Add Image task. We can easily reuse the "no suggestion found" dialog with a slightly different message here.
  • When the user tries to save an Add Image task. Again, we can reuse the existing error dialog here.

Per my comment above, I would like to propose that we check whether the task is valid when on the Suggested edits task queue and show the message, rather than getting it after being taken to the task. It would also be an improvement to the Add links task if we add the check and error message on the Suggested edits page too (and obviously also be more consistent across tasks).

Again, I don't think we could do that without making the user wait more or making the code very complicated.
(Basically, either we'd need some kind of cross-tab communication, which is a complex technical change, or we'd need some kind of server push capability, which is a very complex technical change, or we'd need to query the server and wait for an answer before reacting to a user click/tap, which is not great user experience.)

G. Timezone / Day period
  • The "day" period should be defined as midnight-to-midnight in the user's timezone. - how should this work when the user can make different edits from different time zones? Also, how do we tell the user's time zone? GeoIP? The time zone in their user preferences (which I think defaults to the timezone of the content language, or UTC for more dispersed languages)?

! In T290788#7404700, @kostajh wrote:

It would definitely be less complicated to say "Wait 12 hours" (the "12 hours" string can be updated so when they come back an hour later, it says "11 hours" or whatever), if that is acceptable from a product perspective.

I think it will be easier to understand to keep it as daily, without having to have any countdown clock logic. It also seems like an edge case we could let slip if people are so keen that they make extra edits by going to different timezones (even if someone in Australia changes their timezone back to Pacific time the previous day, they would conceivability only ever be over by 25 max. in a 2-day window). And keeping it simple, using the user preference timezone seems fine.

In practice I think it would mostly happen to people who use some kind of IP-hopping privacy technology. Granted, those tend to be banned from editing anyways.