Files occasionally getting uploaded to Commons without file pages.
Open, HighPublic
Actions

Assigned To

None

Authored By

	Ghouston
	Nov 6 2017, 11:59 PM

Description

Recently, I've noticed (via a bot that I run) some files that have somehow been uploaded to Commons without the creation of corresponding file pages. The first was uploaded on Nov. 1, and I've since found 4 more. Some of the file pages have been created manually, but you can see in the dates that the files were uploaded earlier, by a different user. Here's a list of the files in time order:

https://commons.wikimedia.org/wiki/File:17slemdal_efn.jpg
https://commons.wikimedia.org/wiki/File:Stortinget2017efn.jpg
https://commons.wikimedia.org/wiki/File:Mjstuen2017efn.jpg
https://commons.wikimedia.org/wiki/File:DWCSJ_Hymn.wav
https://commons.wikimedia.org/wiki/File:Ahora_nos_toca_a_nosotros,_ovaci%C3%B3n.jpg

Related Objects

Mentioned In: T245162: Job unable to create file page "Fatal: Call function inNamespace() on null" (via MinervaHooks)
T153565: MediaWiki file operations are fragile, causing occasional data loss
T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Mentioned Here: T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
T187942: Replication lag detection broken in wmf.22

Event Timeline

Ghouston created this task.Nov 6 2017, 11:59 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2017, 11:59 PM

Guanaco awarded a token.Nov 7 2017, 12:51 AM

Hi @Ghouston, thanks for taking the time to report this!
Could you please provide clear steps, click by click, describe how to see the actual problem, and describe what exactly you would expect instead? See https://mediawiki.org/wiki/How_to_report_a_bug for more information.
Do you imply that some problem can be seen on https://commons.wikimedia.org/w/index.php?title=File:Stortinget2017efn.jpg&action=history ?

It can, because the first entry in the history is when the page was created by Guanaco a copule of days after the original upload. Better yet, take a look at https://commons.wikimedia.org/wiki/File:Ahora_nos_toca_a_nosotros,_ovaci%C3%B3n.jpg which doesn't have a proper page and doesn't have a history link at all.

I never saw anything like this prior to 1 Nov. Did something change around that time, a new MediaWiki version or changes to the UploadWIzard?

The problem is currently most obvious if you visit the most recent file: https://commons.wikimedia.org/wiki/File:Ahora_nos_toca_a_nosotros,_ovaci%C3%B3n.jpg

Note that there is no description below the file (the first thing below the picture size is the upload history) and where there would usually be an "Edit" link at the top of the page, there's instead a "Create" link.

On earlier files, the problem is less visible, but for instance if you visit https://commons.wikimedia.org/wiki/File:17slemdal_efn.jpg you'll see that the first version was uploaded by Espen Franck-Nielsen at 19:57, but visiting the history page for the same file: https://commons.wikimedia.org/w/index.php?title=File:17slemdal_efn.jpg&action=history the page was created by Magog the Ogre at 02:11.

Normally, we would expect that when a file is first uploaded, a page is created at the same time, so that the time and user would match between the two histories, and a newly-uploaded file would have an "Edit" link and not a "Create" link.

Based on the complaints by Espen Franck-Nielsen in the files' revision histories, he provided licensing information to the Upload Wizard, so this bug represents a loss of user data somewhere.

[apologies to @Ghouston if I'm treading on his toes]

Thanks, this is helpful information to better understand the problem!

Restricted Application added a project: Multimedia. · View Herald TranscriptNov 7 2017, 12:03 PM

zhuyifei1999 subscribed.Nov 7 2017, 3:05 PM

These five files are apparently the only ones affected by the issue so far – I did some querying to find any more, and by best effort (query below) only finds those five. (Unless there are any instances before October, but that would take a long time to run a query for. Note also that we didn't store 'associated_rev_id' before 2015 or so.)

select *
from commonswiki.logging
where log_type='upload' and log_action='upload'
and log_timestamp>'201710'
and instr(log_params, 'associated_rev_id') = 0
order by log_timestamp desc;

• Ramsey-WMF assigned this task to Cparle.Nov 13 2017, 5:43 PM

• Ramsey-WMF moved this task from Untriaged to Triaged on the Multimedia board.

• Ramsey-WMF added a project: Multimedia-Team-Working-Board.

• Ramsey-WMF moved this task from Triaged to Next up on the Multimedia board.Nov 17 2017, 10:49 PM

Did some investigation on this, but didn't manage to reproduce it. Using @matmarex 's query above (on quarry) I found there are 17 examples of this since August 1 2017 on commons, out of 2.1M uploads logged

Here's what I found

an api call uploads the image from the stash, and the data is recorded in the db by LocalFile::recordUpload2()
this writes to the db, and puts the page creation and writing associated_rev_id to the db into a closure
the closure is executed just before sending the output by MediaWiki::preOutputCommit()

Initially I thought that the closure that creates the page was written to a queue, and an item being dropped from the queue might be the root of the problem. Turns out this is not the case, the closure is not queued but just executed later in the call stack. If anything errors between recording the file data in the db and the execution of the closure it'll prevent the creation of the file page, but I can't think what that 'anything' might be

Cparle removed Cparle as the assignee of this task.Nov 22 2017, 10:35 AM

Cparle subscribed.

Just an example of this from earlier than Nov 1, see

https://commons.wikimedia.org/w/index.php?title=File:Wikimania_Hackathon_2017_07.jpg&action=history (file page created on Aug 14)
https://commons.wikimedia.org/wiki/File:Wikimania_Hackathon_2017_07.jpg (file uploaded on Aug 9)

Cparle removed a project: Multimedia-Team-Working-Board.Nov 22 2017, 10:41 AM

• Ramsey-WMF moved this task from Next up to Triaged on the Multimedia board.Nov 22 2017, 4:02 PM

• Ramsey-WMF moved this task from Incoming to Uploading on the Commons board.

• Ramsey-WMF moved this task from Triaged to Needs Reproduction on the Multimedia board.Nov 24 2017, 5:55 PM

Same on dewiki:
https://de.wikipedia.org/wiki/Datei:Voyaeurs_sur_le_Rhin_1807.JPG

I just found 24 of these files, all uploaded between 2018-02-21T20:27:36 and 2018-02-21T20:28:13 UTC, by various uploaders. Here are links, ordered by upload time:

Here's another batch of 29 from the same day, a few minutes later. The ordering may be random.

There are more, I think at least 400 in total, but I'm not going to list them all here.

Could someone pull a list of files that were my uploads and notify me on-wiki? I just had about 20 or 30 PD Library of Congress images deleted and even listing them is a drag.

Perhelion subscribed.Mar 3 2018, 2:29 PM

This comment was removed by Perhelion.

Perhelion added a comment.Mar 3 2018, 2:33 PM

This comment was removed by Perhelion.

Ankry subscribed.Mar 3 2018, 3:17 PM

Yann subscribed.Mar 4 2018, 3:08 AM

Sixflashphoto subscribed.Mar 4 2018, 5:25 AM

Jcb subscribed.Mar 4 2018, 12:46 PM

In T179884#3892241, @Thgoiter wrote:

Same on dewiki:
https://de.wikipedia.org/wiki/Datei:Voyaeurs_sur_le_Rhin_1807.JPG

This is unlikely to have been the same issue, the file was uploaded in 2015. I think it will be impossible to discover the cause of this problem three years later.

In T179884#4020784, @Fae wrote:

Could someone pull a list of files that were my uploads and notify me on-wiki? I just had about 20 or 30 PD Library of Congress images deleted and even listing them is a drag.

I published a Quarry query with the affected files: https://quarry.wmflabs.org/query/25297 You can find yours by typing "Fæ" into the "Search" box on the right, or clicking the "Download data" button and using your favorite spreadsheet editor. Looks like 182 of the files are yours.

I think the recent surge of 400+ problematic files is also a separate issue. All of them were uploaded on 2018-02-21 (timestamps between 20180221202736 and 20180221203815) and this lines up very nicely with the deployment and revert of MediaWiki 1.31.0-wmf.22, as seen on https://wikitech.wikimedia.org/wiki/Server_Admin_Log#2018-02-21:

20:53 twentyafterfour: MediaWiki Train for 1.31.0-wmf.22 is blocked by T187942

20:39 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.21 (duration: 01m 12s)

20:38 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.21

20:34 twentyafterfour: rolling back group1 to wmf.21

20:28 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.22 (duration: 01m 08s)

20:27 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.22

So, this was almost certainly caused by T187942, and fixed within 11 minutes.

Here's another recent mini-surge:

And the above list is incomplete, see also uploads by user:Joanna_Le between 2018-06-27T19:36 and 2018-06-27T19:48 , eg.
https://commons.wikimedia.org/wiki/File:M._Arcta_s%C5%82ownik_ilustrowany_j%C4%99zyka_polskiego_-_ilustracja_do_has%C5%82a_O%C5%BCypa%C5%82ka.png

It may be caused by T198350, in this latest case.

Aklapper mentioned this in T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment.Jun 28 2018, 12:45 PM

Tgr mentioned this in T153565: MediaWiki file operations are fragile, causing occasional data loss.Jun 28 2018, 3:01 PM

A small batch of 6:

https://commons.wikimedia.org/wiki/File:Starr-170516-8663-Acacia_koa-habit-Waikamoi_Flume_Rd_Olinda-Maui_(35099149301).jpg
https://commons.wikimedia.org/wiki/File:Starr-170516-8668-Dicranopteris_linearis-along_stream-Road_to_Lower_Kula_Pipeline_Haiku_Uka-Maui_(35099166241).jpg
https://commons.wikimedia.org/wiki/File:Starr-170516-8664-Acacia_koa-habit_along_road-Waikamoi_Flume_Rd_Olinda-Maui_(35228949215).jpg
https://commons.wikimedia.org/wiki/File:Starr-170516-0312-Setaria_palmifolia-view_stream-Lower_Kula_Pipeline_Haiku_Uka-Maui_(34419767023).jpg
https://commons.wikimedia.org/wiki/File:Starr-170516-0407-Ulex_europaeus-view_reservoir-Waikamoi_Flume_Rd-Maui_(35063269632).jpg
https://commons.wikimedia.org/wiki/File:Starr-170516-8663-Acacia_koa-habit-Waikamoi_Flume_Rd_Olinda-Maui_(35099149301).jpg

• 4nn1l2 subscribed.Sep 9 2018, 11:31 AM

I still see them occasionally, such as: these two:

https://commons.wikimedia.org/wiki/File:Tekija_(Kladovo).JPG
https://commons.wikimedia.org/wiki/File:Map_detail_showing_piers_and_wharves_on_the_pre-Fire_Seattle_Waterfront_(c.1889).jpg

This is a serious issue. Please fix as soon as possible.

• Marostegui triaged this task as High priority.Jun 20 2019, 5:33 AM

Steinsplitter subscribed.Jun 20 2019, 6:40 AM

Thgoiter awarded a token.Jun 20 2019, 11:26 AM

Finding quite a few lately, such as

https://commons.wikimedia.org/wiki/File:Stanley,_Tasmania.jpg

and these and more from a single uploader:

https://commons.wikimedia.org/wiki/File:Lipovec21.jpg
https://commons.wikimedia.org/wiki/File:Lipovec15.jpg
https://commons.wikimedia.org/wiki/File:Lipovec11.jpg

matmarex mentioned this in T245162: Job unable to create file page "Fatal: Call function inNamespace() on null" (via MinervaHooks).Feb 14 2020, 10:55 PM

AFBorchert subscribed.Jun 15 2020, 9:35 PM

Bawolff subscribed.Feb 23 2024, 1:31 AM

Just checking to see if this is still a thing. Most recent example seems to be https://commons.wikimedia.org/wiki/File:NDL801250_%E5%8B%A7%E6%A5%AD%E5%8A%9F%E7%B8%BE%E9%8C%B2_%E7%AC%AC2%E7%B7%A8_part1.pdf (Feb 29). There seems to be not a whole lot of older images, but possibly that is just because they get fixed or tagged as copyvio and deleted. There is a log entry for that file of an UploadStashFileNotFoundException exception. I'm not really sure how that could happen with the file becoming actually published. Its possible there were multiple publish attempts though and that is from a later one. I'm a bit surprised there are no other log messages. It used the non-async path for assembling, so presumably it was using that path for publishing too.

Files occasionally getting uploaded to Commons without file pages.Open, HighPublicActions

Description

Related Objects

Event Timeline

Files occasionally getting uploaded to Commons without file pages.
Open, HighPublic
Actions