Maniphest T320500

Images being added to the database multiple times
Open, MediumPublicBUG REPORT
Actions

Assigned To

Authored By

	NavinoEvans
	Oct 11 2022, 12:15 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

Eugene233

NavinoEvans

Description

During the process of extracting all images for a campaign, we ignore categories that have already been seen, but do not ignore images that have already been seen.
The only way this error shows up at present is that the count of images is larger than it should be (in cases where duplicate images have been added). But obviously this also increases the load on the database.

Example:
For the following categories with depth=0

Turkey photographs taken on 2020-06-14
Taxis in Ankara

Petscan shows number of files in this combination of categories as 38 files in total, while ISA shows 39.
The extra one image is a duplicate, because there is one image which appears in both categories.

Event Timeline

NavinoEvans created this task.Oct 11 2022, 12:15 PM

NavinoEvans triaged this task as Medium priority.

This should be fixable by adding a check to see if the same image (same page_id and campaign_id) exists at https://gerrit.wikimedia.org/r/plugins/gitiles/labs/tools/Isa/+/refs/heads/master/isa/campaigns/image_updater.py#163.

You could even say that page ids are unique and have a many-to-many relationship between images and campaigns. I don't know exactly how to do this, but I know it's a thing you can do.

NavinoEvans updated the task description. (Show Details)Nov 29 2022, 12:57 PM

Juniorbesong claimed this task.Jan 21 2024, 12:34 AM

Juniorbesong changed the task status from Open to In Progress.Jan 21 2024, 1:04 AM

@Sebastian_Berlin-WMSE, @Eugene233, @NavinoEvans. I wish to suggest on checking if the same image (with the same page_id and campaign_id) exists. Then fetches all images in the given category and if, depth is more than 0, its subcategories filter out pages that do not match for images. What are your reviews?

Change 998432 had a related patch set uploaded (by Juniorbesong; author: Juniorbesong):

[labs/tools/Isa@main] BUG: T320500 modified isa/campaigns/image_updater.py

https://gerrit.wikimedia.org/r/998432

gerritbot added a project: Patch-For-Review.Feb 7 2024, 3:25 PM

Eugene233 moved this task from Incoming Bugs to In progress on the ISA board.Mar 6 2024, 10:18 AM

Eugene233 moved this task from In progress to Review on the ISA board.May 21 2024, 8:27 AM

Resetting task status from "In Progress" to "Open" as this task has been "in progress" for more than one year (see T380300). Feel free to set that status again, or rather break down into smaller subtasks.

Sebastian_Berlin-WMSE unsubscribed.Apr 14 2025, 7:05 AM

Aklapper edited projects, added: Patch-Needs-Improvement; removed: Patch-For-Review.Jan 16 2026, 3:49 PM

Images being added to the database multiple timesOpen, MediumPublicBUG REPORTActions

Description

Event Timeline

Images being added to the database multiple times
Open, MediumPublicBUG REPORT
Actions