Files occasionally getting uploaded to Commons without file pages.
Open, Needs TriagePublic

Description

Recently, I've noticed (via a bot that I run) some files that have somehow been uploaded to Commons without the creation of corresponding file pages. The first was uploaded on Nov. 1, and I've since found 4 more. Some of the file pages have been created manually, but you can see in the dates that the files were uploaded earlier, by a different user. Here's a list of the files in time order:

https://commons.wikimedia.org/wiki/File:17slemdal_efn.jpg
https://commons.wikimedia.org/wiki/File:Stortinget2017efn.jpg
https://commons.wikimedia.org/wiki/File:Mjstuen2017efn.jpg
https://commons.wikimedia.org/wiki/File:DWCSJ_Hymn.wav
https://commons.wikimedia.org/wiki/File:Ahora_nos_toca_a_nosotros,_ovaci%C3%B3n.jpg

Ghouston created this task.Nov 6 2017, 11:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2017, 11:59 PM

Hi @Ghouston, thanks for taking the time to report this!
Could you please provide clear steps, click by click, describe how to see the actual problem, and describe what exactly you would expect instead? See https://mediawiki.org/wiki/How_to_report_a_bug for more information.
Do you imply that some problem can be seen on https://commons.wikimedia.org/w/index.php?title=File:Stortinget2017efn.jpg&action=history ?

It can, because the first entry in the history is when the page was created by Guanaco a copule of days after the original upload. Better yet, take a look at https://commons.wikimedia.org/wiki/File:Ahora_nos_toca_a_nosotros,_ovaci%C3%B3n.jpg which doesn't have a proper page and doesn't have a history link at all.

I never saw anything like this prior to 1 Nov. Did something change around that time, a new MediaWiki version or changes to the UploadWIzard?

bjh21 added a subscriber: bjh21.Nov 7 2017, 11:16 AM

The problem is currently most obvious if you visit the most recent file: https://commons.wikimedia.org/wiki/File:Ahora_nos_toca_a_nosotros,_ovaci%C3%B3n.jpg

Note that there is no description below the file (the first thing below the picture size is the upload history) and where there would usually be an "Edit" link at the top of the page, there's instead a "Create" link.

On earlier files, the problem is less visible, but for instance if you visit https://commons.wikimedia.org/wiki/File:17slemdal_efn.jpg you'll see that the first version was uploaded by Espen Franck-Nielsen at 19:57, but visiting the history page for the same file: https://commons.wikimedia.org/w/index.php?title=File:17slemdal_efn.jpg&action=history the page was created by Magog the Ogre at 02:11.

Normally, we would expect that when a file is first uploaded, a page is created at the same time, so that the time and user would match between the two histories, and a newly-uploaded file would have an "Edit" link and not a "Create" link.

Based on the complaints by Espen Franck-Nielsen in the files' revision histories, he provided licensing information to the Upload Wizard, so this bug represents a loss of user data somewhere.

[apologies to @Ghouston if I'm treading on his toes]

Thanks, this is helpful information to better understand the problem!

Restricted Application added a project: Multimedia. · View Herald TranscriptNov 7 2017, 12:03 PM

These five files are apparently the only ones affected by the issue so far – I did some querying to find any more, and by best effort (query below) only finds those five. (Unless there are any instances before October, but that would take a long time to run a query for. Note also that we didn't store 'associated_rev_id' before 2015 or so.)

select *
from commonswiki.logging
where log_type='upload' and log_action='upload'
and log_timestamp>'201710'
and instr(log_params, 'associated_rev_id') = 0
order by log_timestamp desc;
Ramsey-WMF moved this task from Untriaged to Triaged on the Multimedia board.Nov 13 2017, 5:43 PM
Ramsey-WMF assigned this task to Cparle.
Ramsey-WMF moved this task from Triaged to Next up on the Multimedia board.Nov 17 2017, 10:49 PM

Did some investigation on this, but didn't manage to reproduce it. Using @matmarex 's query above (on quarry) I found there are 17 examples of this since August 1 2017 on commons, out of 2.1M uploads logged

Here's what I found

  • an api call uploads the image from the stash, and the data is recorded in the db by LocalFile::recordUpload2()
  • this writes to the db, and puts the page creation and writing associated_rev_id to the db into a closure
  • the closure is executed just before sending the output by MediaWiki::preOutputCommit()

Initially I thought that the closure that creates the page was written to a queue, and an item being dropped from the queue might be the root of the problem. Turns out this is not the case, the closure is not queued but just executed later in the call stack. If anything errors between recording the file data in the db and the execution of the closure it'll prevent the creation of the file page, but I can't think what that 'anything' might be

Cparle removed Cparle as the assignee of this task.Nov 22 2017, 10:35 AM
Cparle added a subscriber: Cparle.
Ramsey-WMF moved this task from Next up to Triaged on the Multimedia board.Nov 22 2017, 4:02 PM
Ramsey-WMF moved this task from Incoming to Uploading on the Commons board.

I just found 24 of these files, all uploaded between 2018-02-21T20:27:36 and 2018-02-21T20:28:13 UTC, by various uploaders. Here are links, ordered by upload time:

https://commons.wikimedia.org/wiki/File:Chinatown,_New_York_City_-_Chinese_New_Year_LCCN2001704350.jpg
https://commons.wikimedia.org/wiki/File:78_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11127096225).jpg
https://commons.wikimedia.org/wiki/File:ЧС7-283,_Россия,_Москва,_депо_Москва-Пассажирская-Курская_(Trainpix_190612).jpg
https://commons.wikimedia.org/wiki/File:585_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11271523756).jpg
https://commons.wikimedia.org/wiki/File:577_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11271185945).jpg
https://commons.wikimedia.org/wiki/File:ЧС7-310,_Украина,_Днепропетровская_область,_депо_Днепропетровск_(Trainpix_190611).jpg
https://commons.wikimedia.org/wiki/File:Père-Lachaise_-_Division_4_-_Legrand_01.jpg
https://commons.wikimedia.org/wiki/File:513_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11125628125).jpg
https://commons.wikimedia.org/wiki/File:49_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11121484414).jpg
https://commons.wikimedia.org/wiki/File:Chinatown,_New_York_City_LCCN2001704349.tif
https://commons.wikimedia.org/wiki/File:480_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11125861765).jpg
https://commons.wikimedia.org/wiki/File:ЧС7-131,_Украина,_Харьковская_область,_депо_Октябрь_(Trainpix_190610).jpg
https://commons.wikimedia.org/wiki/File:41_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11269171423).jpg
https://commons.wikimedia.org/wiki/File:554_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11125492436).jpg
https://commons.wikimedia.org/wiki/File:G05A1018.jpg
https://commons.wikimedia.org/wiki/File:Verkeer_rond_Gare_Saint-Lazare,_Bestanddeelnr_254-0644.jpg
https://commons.wikimedia.org/wiki/File:Père-Lachaise_-_Division_4_-_Leconte,_Villenfin_&_Carpentier_01.jpg
https://commons.wikimedia.org/wiki/File:Nat'l_Woman's_Party_LOC_hec.14179.jpg
https://commons.wikimedia.org/wiki/File:550_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11123093776).jpg
https://commons.wikimedia.org/wiki/File:507_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11125394174).jpg
https://commons.wikimedia.org/wiki/File:KZ4AT-0010,_Казахстан,_Карагандинская_область,_перегон_Караганда_-_Караганозек_(Trainpix_190604).jpg
https://commons.wikimedia.org/wiki/File:Chinatown,_New_York_City_LCCN2001704349.jpg
https://commons.wikimedia.org/wiki/File:583_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11121384543).jpg
https://commons.wikimedia.org/wiki/File:390_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11123310966).jpg

Here's another batch of 29 from the same day, a few minutes later. The ordering may be random.

https://commons.wikimedia.org/wiki/File:Draglines,_persbuizen,_zandzuigers,_ruilverkaveling_nieuwepolder,_Bestanddeelnr_165-0233.jpg
https://commons.wikimedia.org/wiki/File:Nat'l_Woman's_Party_LOC_hec.14179.tif
https://commons.wikimedia.org/wiki/File:139_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11271494566).jpg
https://commons.wikimedia.org/wiki/File:Père-Lachaise_-_Division_4_-_Lubin-Lejeune_02.jpg
https://commons.wikimedia.org/wiki/File:84_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11125859826).jpg
https://commons.wikimedia.org/wiki/File:71_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11123013385).jpg
https://commons.wikimedia.org/wiki/File:Treasury_Dept._lab_LOC_hec.14167.tif
https://commons.wikimedia.org/wiki/File:O'ZBEKISTON-0002,_Узбекистан,_Ташкент,_станция_Тукимачи_(Trainpix_190639).jpg
https://commons.wikimedia.org/wiki/File:Nat'l_Woman's_Party_LOC_hec.14180.jpg
https://commons.wikimedia.org/wiki/File:115_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11122832855).jpg
https://commons.wikimedia.org/wiki/File:Père-Lachaise_-_Division_4_-_Lubin-Lejeune_01.jpg
https://commons.wikimedia.org/wiki/File:573_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11123208755).jpg
https://commons.wikimedia.org/wiki/File:Chinatown,_New_York_City_-_New_Years_feast,_Jan._26,_1906_LCCN2001704351.jpg
https://commons.wikimedia.org/wiki/File:ЭП2К-279,_Russia,_Chelyabinsk_region,_Khrebet_-_Urzhumka_stretch_(Trainpix_190627).jpg
https://commons.wikimedia.org/wiki/File:106_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11127957413).jpg
https://commons.wikimedia.org/wiki/File:103_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11126295636).jpg
https://commons.wikimedia.org/wiki/File:Père-Lachaise_-_Division_4_-_Legrand_03.jpg
https://commons.wikimedia.org/wiki/File:Agent_regelt_het_verkeer,_Bestanddeelnr_254-0643.jpg
https://commons.wikimedia.org/wiki/File:ВЛ80С-214,_Казахстан,_Карагандинская_область,_перегон_Караганда_-_Караганозек_(Trainpix_190625).jpg
https://commons.wikimedia.org/wiki/File:Chinatown,_New_York_City_-_Chinese_New_Year_LCCN2001704350.tif
https://commons.wikimedia.org/wiki/File:101_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11126687444).jpg
https://commons.wikimedia.org/wiki/File:Burgemeester_Samkalden_opent_winkelcentrum_Gelderlandplein_in_Buiten_Veldert_het,_Bestanddeelnr_921-9024.jpg
https://commons.wikimedia.org/wiki/File:42_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11124697095).jpg
https://commons.wikimedia.org/wiki/File:O'Z-Y0102,_Узбекистан,_Ташкент,_станция_Тукимачи_(Trainpix_190622).jpg
https://commons.wikimedia.org/wiki/File:97_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11270075195).jpg
https://commons.wikimedia.org/wiki/File:36_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11123184574).jpg
https://commons.wikimedia.org/wiki/File:Père-Lachaise_-_Division_4_-_Legrand_02.jpg
https://commons.wikimedia.org/wiki/File:92_of_'The_Earth_and_its_Inhabitants._The_European_section_of_the_Universal_Geography_by_E._Reclus._Edited_by_E._G._Ravenstein._Illustrated_by_..._engravings_and_maps'_(11122813433).jpg
https://commons.wikimedia.org/wiki/File:Moslims_met_schapen_bij_het_slachthuis_in_Weesp,_Bestanddeelnr_934-4821.jpg
https://commons.wikimedia.org/wiki/File:Draglines,_persbuizen,_zandzuigers,_bruggen,_ruilverkaveling_nieuwepolder,_Bestanddeelnr_165-0232.jpg

There are more, I think at least 400 in total, but I'm not going to list them all here.

Fae added a subscriber: Fae.Mar 3 2018, 2:20 PM

Could someone pull a list of files that were my uploads and notify me on-wiki? I just had about 20 or 30 PD Library of Congress images deleted and even listing them is a drag.

This comment was removed by Perhelion.
This comment was removed by Perhelion.
Ankry added a subscriber: Ankry.Mar 3 2018, 3:17 PM
Yann added a subscriber: Yann.Mar 4 2018, 3:08 AM
Jcb added a subscriber: Jcb.Mar 4 2018, 12:46 PM

This is unlikely to have been the same issue, the file was uploaded in 2015. I think it will be impossible to discover the cause of this problem three years later.

Could someone pull a list of files that were my uploads and notify me on-wiki? I just had about 20 or 30 PD Library of Congress images deleted and even listing them is a drag.

I published a Quarry query with the affected files: https://quarry.wmflabs.org/query/25297 You can find yours by typing "Fæ" into the "Search" box on the right, or clicking the "Download data" button and using your favorite spreadsheet editor. Looks like 182 of the files are yours.

I think the recent surge of 400+ problematic files is also a separate issue. All of them were uploaded on 2018-02-21 (timestamps between 20180221202736 and 20180221203815) and this lines up very nicely with the deployment and revert of MediaWiki 1.31.0-wmf.22, as seen on https://wikitech.wikimedia.org/wiki/Server_Admin_Log#2018-02-21:

  • 20:53 twentyafterfour: MediaWiki Train for 1.31.0-wmf.22 is blocked by T187942
  • 20:39 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.21 (duration: 01m 12s)
  • 20:38 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.21
  • 20:34 twentyafterfour: rolling back group1 to wmf.21
  • 20:28 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.22 (duration: 01m 08s)
  • 20:27 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.22

So, this was almost certainly caused by T187942, and fixed within 11 minutes.