Page MenuHomePhabricator
Create Task
Maniphest T17889

Write and run script to find non-existent images on Wikimedia wikis
Open, LowPublicFeature
Actions

Description

Currently on Commons and on the various projects, there are images that have description pages but their images have disappeared. Because the meta data (MIME type, file size, directory, etc.) is stored in the database, it's not easily possible to find which images have this issue. (Without a definitive list of broken images, it also makes it very difficult to know whether this is a growing problem somehow related to WMF's servers or not.)

It would be nice if someone were to write a script that checked each image in a database to ensure that it exists and that it is not 0 bytes. Example (though this will likely be deleted by an admin at some point): http://commons.wikimedia.org/wiki/Image:Hatogayacity_Fire_Department.jpg

Version: unspecified
Severity: enhancement

Details

Reference
bz15889

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenFeatureNoneT18660 Database table cleanup (tracking)
 OpenFeatureNoneT17889 Write and run script to find non-existent images on Wikimedia wikis

Event Timeline

• bzimport raised the priority of this task from to Low.Nov 21 2014, 10:19 PM
• bzimport added a project: Wikimedia-Site-requests.
• bzimport set Reference to bz15889.
• bzimport added a subscriber: Unknown Object (MLST).
• MZMcBride created this task.Oct 7 2008, 9:17 PM
Peachey88 added a comment.Jan 16 2012, 10:46 AM

Is this still a issue that needs to be investigated?

Since you are a TS wizz, Can this be done with the recent DB metadata improvements?

Krenair added a comment.Jun 10 2012, 5:54 PM

Bumping this. Still an issue?

Krenair added a comment.Jul 26 2012, 1:10 AM

Hello?

Withoutaname added a comment.May 14 2014, 7:48 AM

Are we talking about:

  1. Running cleanupImages.php? Would probably need shell.
  2. Running a database query? It would be something like "SELECT page_title FROM page WHERE NOT EXISTS (SELECT img_name FROM image WHERE img_name = page_title) AND NOT EXISTS (SELECT img_name FROM commonswiki_p.image WHERE img_name = page_title) AND page_namespace = 6 AND page_is_redirect = 0 LIMIT 1000;"
  3. Fixing bug 32551?
• MZMcBride added a comment.May 17 2014, 12:52 AM

(In reply to TeleComNasSprVen from comment #4)

Are we talking about:

  1. Running cleanupImages.php? Would probably need shell.
  2. Running a database query? It would be something like "SELECT page_title

FROM page WHERE NOT EXISTS (SELECT img_name FROM image WHERE img_name =
page_title) AND NOT EXISTS (SELECT img_name FROM commonswiki_p.image WHERE
img_name = page_title) AND page_namespace = 6 AND page_is_redirect = 0 LIMIT
1000;"

  1. Fixing bug 32551?

The bug summary is "Write and run script to find non-existent images on Wikimedia wikis". We're probably talking about that.

Peachey88 unsubscribed.Aug 9 2015, 9:30 AM
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptAug 9 2015, 9:30 AM
Peachey88 added a project: Multimedia.Aug 9 2015, 9:30 AM
Peachey88 set Security to None.
Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 5:51 PM
Restricted Application added a subscriber: JEumerus. · View Herald TranscriptJan 26 2016, 12:45 PM
Meno25 unsubscribed.Feb 22 2016, 6:06 PM
Dereckson subscribed.Sep 9 2016, 2:51 AM
In T17889#199145, @Withoutaname wrote:

Are we talking about:
[…]

  1. Fixing bug 32551?

T34551 has been resolved.

Dereckson assigned this task to • MZMcBride.EditedSep 9 2016, 2:53 AM
Dereckson moved this task from Backlog to Analysis / under discussion on the Wikimedia-Site-requests board.

[ Assigning this to original poster, to clarify the requested action and the goal per previous comment. Once done, feel free to deassign for further triaging. ]

Dereckson raised the priority of this task from Low to Needs Triage.Sep 9 2016, 2:54 AM
• MZMcBride added a comment.Sep 9 2016, 5:44 AM

The idea here was to have a script iterate over all known images in a MediaWiki wiki and check that the files actually exist on the file system and are accessible and are not corrupt. A maintenance script that checks every image's SHA1 or file size from the file system directly and compares to the information we have stored in the image database table would resolve this task.

It's been many years since this task was filed. It may now be moot.

Dereckson added a comment.Sep 9 2016, 5:59 AM

That could be combined with T32961.

Dereckson added a subscriber: matmarex.Sep 9 2016, 5:59 AM
matmarex added a comment.Oct 10 2016, 9:19 PM

maintenance/findMissingFiles.php already exists. I'd rather not combine T32961 with anything else.

MarcoAurelio added a project: MediaWiki-Maintenance-system.Nov 22 2017, 10:07 AM
Jayprakash12345 subscribed.Nov 22 2017, 2:13 PM
DannyS712 moved this task from Backlog to Requested scripts on the MediaWiki-Maintenance-system board.Nov 28 2019, 2:02 AM
Krinkle removed a project: MediaWiki-Maintenance-system.Apr 3 2020, 3:11 PM
Krinkle removed a subscriber: • wikibugs-l-list.
Aklapper removed • MZMcBride as the assignee of this task.Jun 19 2020, 4:17 PM

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

CptViraj subscribed.Aug 30 2020, 4:25 PM
Aklapper added a comment.Oct 12 2020, 5:37 AM
In T17889#2704464, @matmarex wrote:

maintenance/findMissingFiles.php already exists.

...which makes me wonder what else is still wanted in this task.

matmarex added a comment.Oct 12 2020, 2:42 PM

Presumably, the "run" part.

Urbanecm subscribed.Oct 12 2020, 4:46 PM

I ran this script at testwiki, to see how it works:

[urbanecm@mwmaint2001 ~]$ time mwscript findMissingFiles.php --wiki=testwiki
mwstore://local-multiwrite/local-public/archive/a/a3/20090409180603!AnyNonsense.png
mwstore://local-multiwrite/local-public/archive/4/47/20120830220223!Liberty_Bell_slot_machine_2012-07-30_13-52-19.jpg
mwstore://local-multiwrite/local-public/archive/5/51/20170823084259!SIPIJellyBeans.jpg
mwstore://local-multiwrite/local-public/archive/9/97/20200609104426!Test-by-nokib2.jpg
mwstore://local-multiwrite/local-public/archive/9/9c/20200716100612!Test_image_copyright.jpg

real    0m27.704s
user    0m7.204s
sys     0m1.756s
[urbanecm@mwmaint2001 ~]$

Can we confirm it yields the expected output before running it at more wikis?

matmarex unsubscribed.Oct 12 2020, 5:53 PM
RhinosF1 subscribed.Nov 10 2020, 7:52 AM
Aklapper triaged this task as Low priority.Feb 4 2022, 10:51 AM
Aklapper changed the subtype of this task from "Task" to "Feature Request".
Stang subscribed.Mar 28 2022, 2:25 PM
Bugreporter merged a task: T182822: Generate a list of files that are supposed to exist but 404s.Feb 12 2024, 3:02 PM
Bugreporter added subscribers: zhuyifei1999, jcrespo, ArielGlenn and 8 others.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL