Page MenuHomePhabricator

Add an Image: Invalidate accepted suggestions
Closed, ResolvedPublic2 Estimated Story Points

Description

I was informed of a case where one out-of-topic image was added 3 times to the same article.

Why:
Prior to this patch, AddImageSubmissionHandler invalidated only
(certain) rejected suggestions. This only makes difference when
the accepting edit is reverted by another user, as then the image
is in unillustrated state with no feedback passed to the pipeline,
resulting in the suggestion staying in the task pool, without anything
removing it.

What:
Fix the condition in AddImageSubmissionHandler::handle, so that
it invalidates accepted submissions as well.

Acceptance Criteria:

Given an image suggestion is accepted but later removed from the article,
When the image suggestion task pool is populated,
Then the same image should never be suggested again

Event Timeline

Urbanecm_WMF triaged this task as High priority.
Urbanecm_WMF subscribed.

I'll do a first investigation. Accepting a suggestion should remove it from the pool, which is not happening for some reason.

I investigated the issue, and I noticed that there are multiple users evaluating a suggestion on the page, all of them decided to accept the image (logs). All edits are reasonably far from each other (months), which means the search index had plenty of time to update itself, even if the cache mechanism in AddImageSubmissionHandler::invalidateRecommendation was broken. As such, we're (likely) not invalidating the suggestion at all.

To verify that, I checked the event.mediawiki_image_suggestions_feedback table in Hadoop, where we store all invalidated recommendations (so that the suggestions pipeline can exclude them from being suggested again). A SQL query for frwiki invalidations done on the day of October 15, 2023 (when the last problematic edit was made) shows the following:

hive (event)> select wiki, dt, page_id, filename, origin_wiki, is_accepted from mediawiki_image_suggestions_feedback where year=2023 and month=10 and day=15 and wiki='frwiki' limit 30;                                                        
OK
wiki    dt      page_id filename        origin_wiki     is_accepted
frwiki  2023-10-15T13:25:36Z    4270601 Badal_Pillar_Inscription.jpg    commonswiki     false
frwiki  2023-10-15T14:18:09Z    1191789         commonswiki     false
frwiki  2023-10-15T14:16:28Z    4184537 Portulaca_F1.jpg        commonswiki     false
frwiki  2023-10-15T15:38:59Z    6080938 Charging_stations_in_SF_City_Hall_02_2009_02.jpg        commonswiki     false
frwiki  2023-10-15T17:53:53Z    816122  James-clerk-maxwell3.jpg        commonswiki     false
frwiki  2023-10-15T09:37:00Z    1766784 Tchaikovsky_1906_Evans.PNG      commonswiki     false
frwiki  2023-10-15T01:02:35Z    4381867         commonswiki     false
frwiki  2023-10-15T14:49:00Z    5095528         commonswiki     false
frwiki  2023-10-15T15:42:23Z    8483115         commonswiki     false
frwiki  2023-10-15T15:30:14Z    5095528         commonswiki     false
frwiki  2023-10-15T15:11:02Z    5095528         commonswiki     false
frwiki  2023-10-15T15:36:22Z    8483115         commonswiki     false
frwiki  2023-10-15T17:31:46Z    210203          commonswiki     false
frwiki  2023-10-15T19:34:11Z    6916254         commonswiki     false
frwiki  2023-10-15T19:34:10Z    10442157                commonswiki     false
frwiki  2023-10-15T06:40:31Z    8265981 Sommerfest_2011_der_SPÖ_(5883927870).jpg        commonswiki     false
frwiki  2023-10-15T08:18:24Z    174705          commonswiki     false
frwiki  2023-10-15T08:14:59Z    174705          commonswiki     false
frwiki  2023-10-15T09:37:47Z    957182          commonswiki     false
Time taken: 0.35 seconds, Fetched: 19 row(s)
hive (event)>

Most importantly, neither the image (Powerful_number_Cuisenaire_rods_9.png) nor the page ID for the problematic page (168152, cf. page info) is present in the table. This confirms the image was not invalidated at all, and thus is being re-recommended to frwiki newcomers, and from time to time, added into the article.

Next, I checked the invalidation chain to see why this might be. We do invalidation in three places:

  1. Action API action=growthinvalidateimagerecommendation
  2. REST API /growthexperiments/v0/suggestions/addimage/feedback/{title}
  3. In VisualEditorHooks::onVisualEditorApiVisualEditorEditPostSave

For successful edits, we use the third option, VisualEditorHooks::onVisualEditorApiVisualEditorEditPostSave, which calls SubmissionHandler::handle() on the submission handler for the given task type. In the case of Add Image edits, this results to calling AddImageSubmissionHandler::handle.

It turns out that AddImageSubmissionHandler::handle includes the following conditions:

// Remove this image from being recommended in the future, unless it was rejected with
// one of the "not sure" options.
if ( array_diff( $reasons, self::REJECTION_REASONS_UNDECIDED ) ) {
	$this->invalidateRecommendation(
		$taskType,
		$page,
		$user->getId(),
		$accepted,
		$filename,
		$sectionTitle,
		$sectionNumber,
		$reasons
	);
}

This is supposed to invalidate all submissions, except when it was rejected because the user is unsure. However, what that code actually does is that it invalidates all rejected submissions, except when they were rejected because the user is unsure, leaving the accepted submissions within the task pool (array_diff( [], ... ) always evaluates to an empty array, which is a false-ish value in PHP). This is normally not visible (except when the edit gets reverted), as the suggestions pipeline would exclude the image during its next run because "this article already has an image". Because of the revert, the suggestion is being left inside the pool with nothing getting it out of it.

There is an easy fix: fix the condition to invalidate both accepted and rejected submissions. Moving to sprint!

Change 972017 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] AddImageSubmissionHandler: Invalidate accepted suggestions

https://gerrit.wikimedia.org/r/972017

KStoller-WMF renamed this task from Provide a way for communities to exclude images from being suggested to Add an Image: Invalidate accepted suggestions .Nov 6 2023, 9:05 PM
KStoller-WMF updated the task description. (Show Details)

Change 972017 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] AddImageSubmissionHandler: Invalidate accepted suggestions

https://gerrit.wikimedia.org/r/972017

Urbanecm_WMF set the point value for this task to 2.Nov 8 2023, 3:55 PM

@Urbanecm_WMF - I tested it in beta. The test case steps

  • user A added a suggested image
  • user B removed the added image, i.e. clicks Undo in View history.

Result: the article is still suggested with the same image.

Additional testing:
Rejected images get removed from the suggestions. Images that are not accepted with "Not sure" don't get removed.

On the other hand, the articles with accepted images are not deleted from the suggestions, e.g. cswiki beta still lists the following article https://cs.wikipedia.beta.wmflabs.org/w/index.php?title=Dohl%C3%AD%C5%BEet_a_trestat&action=history as a suggestion for "Add an image to an unillustrated article". The article suggestion should be removed immediately after accepting the image. Or it's just deficiencies of beta environment?

Hi @Etonkovidova, thanks for the tests. As T306349: Public-facing API for querying image suggestion recommendations and submitting user feedback is not resolved, we currently do not have a publicly available API for the image recommendation service. In other words, the current image recommendation API can be only called from Wikimedia production, and it is not available in Wikimedia cloud (including the beta sites). To make the image recommendation feature at least somehow testable in beta, we mirror production recommendations in beta (if Foo article has an image recommendation in production enwiki, the Foo article in beta enwiki will have the same image recommended). If you accept an image in production for one article, the same article in beta should lose the recommendation as well.

TLDR: Because of T306349, this issue is only testable in production, and cannot be tested in beta.

Hi @Etonkovidova, thanks for the tests. As T306349: Public-facing API for querying image suggestion recommendations and submitting user feedback is not resolved, we currently do not have a publicly available API for the image recommendation service. In other words, the current image recommendation API can be only called from Wikimedia production, and it is not available in Wikimedia cloud (including the beta sites). To make the image recommendation feature at least somehow testable in beta, we mirror production recommendations in beta (if Foo article has an image recommendation in production enwiki, the Foo article in beta enwiki will have the same image recommended). If you accept an image in production for one article, the same article in beta should lose the recommendation as well.

TLDR: Because of T306349, this issue is only testable in production, and cannot be tested in beta.

Thanks, @Urbanecm_WMF! Since the main scope of the task is done, the task is closed as Resolved.