@Majora I want this.

In T234192#5578317, @Bawolff wrote:

One minor thing to note, that sometimes this description is used to run stats on how popular different upload methods are. However, that would be a silly reason to keep it the way it is. I've always been kind of annoyed by the lack of descriptive description.

In T234192#5567124, @Aklapper wrote:

@Masumrezarock100: I don't see general agreement yet how exactly to fix this task (there are several suggestions in the task description), so I don't think this task is ready to receive a patch yet...? I might misinterpret though.

https://commons.wikimedia.org/wiki/File:Busan_tower_by_night.jpg First four revisions missing!

I haven't read all this, but could https://commons.wikimedia.org/w/index.php?title=Commons:Administrators%27_noticeboard&oldid=370199209#A_contribution_has_been_deleted?_(that,_or_database_error?) be related?

@DannyS712 perhaps you could make the example above into something that would actually work? Also, @Fae asked if it would be possible to add information from the EXIF.

In T234192#5533564, @Aklapper wrote:

I did not say that "community consensus is required". I referred to some consensus / feedback which specific changes make sense and should be made. :)

In T234192#5533383, @Aklapper wrote:

Once there is agreement, you could propose a patch in Gerrit by changing that string in https://phabricator.wikimedia.org/diffusion/EUWI/browse/master/resources/mw.UploadWizardDetails.js$973

When uploading the first revision of https://commons.wikimedia.org/wiki/File:Stek5.jpg (not deleted, just overwritten), I now get no error.

With 4K video becoming more and more commonplace, 4GB isn't always enough. https://commons.wikimedia.org/wiki/File:Politparade.webm and https://commons.wikimedia.org/wiki/File:Genderwahn.webm are encoded to fit within MediaWiki's limit, which isn't a good sign. And 9 Mbps for 4K.. Well.. Could be worse, but.. not ideal. And they lack 1440p and 2160p transcodes, presumably due to the file size limit.

In T20112#225672, @brion wrote:

It would probably be good in general to have an additional structured metadata area for this sort of thing... and license info... :P

In T232756#5489085, @Nemo_bis wrote:

As written, this is not a MediaWiki core issue: the "confirmed" group is a Wikimedia configuration.
https://noc.wikimedia.org/conf/highlight.php?file=CommonSettings.php

// Confirmed can do anything autoconfirmed can.
// T213003: this should happen after all the extensions had been loaded, but
// before their extension functions in case they're relying on permissions.
array_unshift( $wgExtensionFunctions, function () {
	global $wgGroupPermissions;
	$wgGroupPermissions['confirmed'] = $wgGroupPermissions['autoconfirmed'];
	$wgGroupPermissions['confirmed']['skipcaptcha'] = true;
} );

If the request is to add a variable like $wgAutodemoteOnce with additional possibilities beyond those of https://www.mediawiki.org/wiki/Manual:$wgAutopromoteOnce , we may need a stronger use case.

In T214230#5482840, @Strainu wrote:

In T214230#5482731, @AlexisJazz wrote:

Enabling a feature that causes this much of a mess is not a solution for anything.

Please try to see this from a casual user's PoV: with this feature they doesn't need to worry about where is the picture, how to upload it, how to come back from commons to the article etc. They have a seamless workflow, hopefully in their native language, that makes the picture instantly usable in their article. If we disable it, they'll have to go through another help page explaining how to upload pictures etc. which would decrease the chance of a successful edit even more. This is the problem for which cross-wiki uploads are the solution.
I'm not denying this creates problems for wiki maintainers, and I encourage all of you to log bugs and feature requests against it and push for them to be prioritized, but disabling the whole thing is not the right step forward for our projects.

In T214230#5456613, @Strainu wrote:

Disabling cross-wiki uploads instead of working on new tools to help curate and cleanup those files is a huge step backwards for the good faith newbies who want to contribute pictures to articles.
I am aware of the discussions regarding the lack of ownership - wiki admins do not see the file and Commons admins are overworked, but the solution is to develop new tools to curate these uploads rather than just disabling them.
Very likely, temporarily would be for years in this case.

Disabling crosswiki uploads permanently would be a relief..

@Urbanecm Arjoopy got an error message about 900 edits per 180 seconds.

I don't know if I even support this myself anymore.. The only thing that should be fixed is the error message. @DannyS712, did you already create a task for that?

(this would be in addition to 100 moves/minute for file movers)

In T228317#5342748, @Jdforrester-WMF wrote:

In T228317#5342493, @AlexisJazz wrote:

@DannyS712 no, currently not. But the rate limit has one purpose: to curb vandalism without limiting any ordinary use. Limiting vetted users to the degree that it hampers their work means it's not configured correctly.

This is not entirely true. The main purpose is to maintain the stability of the servers, with the (very welcome but) secondary purpose of inhibiting vandalism until a sysop can block them. That means that well-meaning community members may from time to time bump into one of these limits.

@Urbanecm Thanks, I understand it now, the configuration is on top of the defaults.

@DannyS712 that's on top of the MediaWiki defaults? MediaWiki default is 8 moves per minute AFAIK, which would seem plausible if the system was counting Arjoopy's moves over 2 minutes.

@DannyS712 no, currently not. But the rate limit has one purpose: to curb vandalism without limiting any ordinary use. Limiting vetted users to the degree that it hampers their work means it's not configured correctly.

@Nirmos "on any Wikimedia project I know of", you don't know Commons, do you? This is normal. According to https://commons.wikimedia.org/wiki/Commons:Village_pump/Proposals/Archive/2018/06#Rate_limit_is_at_90_edits_per_minute the ratelimit on Commons is set at 10500 per 3 minutes for patrolled/autopatrolled/image reviewers. (not sure if this is still the current setting.. I thought I ran into it again some time ago)

API request failed (readonly): The wiki is currently in read-only mode. <i>at Sat, 13 Jul 2019 22:21:04 GMT</i> <u>served by mw1284</u> (for https://commons.wikimedia.org/wiki/File:10_Yuan_-_Bank_of_Honan_Province_(1922)_02.jpg )

Also requested at https://commons.wikimedia.org/wiki/MediaWiki_talk:Gadget-AjaxQuickDelete.js#Edit_request. @Perhelion, @Rillke ?

@Taketa this happens when you try to restore a version with different structured data. (depicts or captions) If no structured data was altered, you get the plain text.

@Aklapper I changed the title again because it turns out to be a general issue.

@Urbanecm in spirit, the vote was to determine the maximum allowable number. The outcome of the vote is either "unlimited" or "a finite number" depending on how much support you want. One million is a also a finite number, so it's the same in practice. So if UploadWizard would reliably start supporting a higher number in the future (who knows, could happen), autopatrolled users should get that. (this future-proofs the proposal) I imagine that higher number would be connected to the mass-upload flag anyway. At this moment, the technical limit is 500 so that's what it'll be. Anyone looking to upload more will probably prefer third party tools anyway.

@DannyS712 thanks for the quick response! Perhaps move the "T214003" comment to the upload_by_url line, but that's just nitpicking.

I now got the same error when trying to upload https://upload.wikimedia.org/wikipedia/commons/archive/f/f8/20190615181201%21Stek5.jpg with UploadWizard, the first revision of https://commons.wikimedia.org/wiki/File:Stek5.jpg or https://upload.wikimedia.org/wikipedia/commons/archive/6/6e/20190618200102%21Sakuragicho_Station_Name_Board_Pikachu.jpg, the first revision of https://commons.wikimedia.org/wiki/File:Sakuragicho_Station_Name_Board_Pikachu.jpg.

The file in question. (unknown if copyvio or not, do not use for purposes other than testing)

https://commons.wikimedia.org/wiki/File:President_Lula_and_Marisa.jpg first two revisions missing.

In T27707#4922020, @brion wrote:

I'll do a more thorough investigation on removing them later on once I've got local XP & Vista instances to test with. Sounds like there may be more inconsistencies in terms of later changes in service packs that makes it hard to confirm in testing!

The Naughty.png exploit works on Windows 2000 SP4 with IE5. (I can't be arsed to install IE6 on it) So does ietest-htmlsig.jpg. ietest64-jpgsig.jpg does not work however, so indeed there may be a difference between jpg and png. This confirms that even Windows XP isn't vulnerable, provided you have SP3. (earlier service packs not tested)

@brion What's the status? I wanted to upload https://www.flickr.com/photos/boellstiftung/25524909051/ but couldn't.

In T213484#5223408, @Aklapper wrote:

Is this a duplicate of T31284: Upload form should change file extensions to the canonical form automatically (lowercase, jpeg→jpg etc.)? If it is not, what is the difference exactly?

In T27707#4922020, @brion wrote:

In T27707#4922017, @AlexisJazz wrote:

https://www.flickr.com/photos/tinto/30943950124/ and other photos from this Flickr user.

Perfect! Confirmed that the original file there fails to upload on test.wikipedia.org but uploads intact on my local instance with the patch. :D

I loaded the "naughty.png" on IE6 served as image/png. I see a fox. Perhaps your webserver was misconfigured in 2005?

I assure you I tested quite thoroughly at the time, and several people have independently discovered and reported the same vulnerability over the years. Here's another description, which includes an independently written sample PNG-containing-HTML exploit as well: https://forums.hak5.org/topic/6565-xss-exploit-in-ie-by-design-says-microsoft/
In any case, we can put this part on the back burner since keeping the exact reverse-engineered IE security checks doesn't seem to block JPEG files with exif data the same way the more expansive checks did... I'll do a more thorough investigation on removing them later on once I've got local XP & Vista instances to test with. Sounds like there may be more inconsistencies in terms of later changes in service packs that makes it hard to confirm in testing!

In T215128#4923631, @zhuyifei1999 wrote:

I upgraded youtube_dl on all hosts. Could you try again?
Anyhow, this task belongs to https://github.com/toolforge/video2commons/issues, not phabricator.

In T27707#4921905, @brion wrote:

The above patch https://gerrit.wikimedia.org/r/487527 removes some of the non-scripting tags from the checks in UploadBase::detectScript, and makes the conservative/exact IE heuristics called from UploadBase::verifyMimeType optional (but still on by default). This also deprecates $wgAllowTitleInSVG since <title is no longer looked for.
I think this should get JPEGs and such with links in their EXIF metadata working; are there some samples of files that were tripping up before that we can use to add in test cases? A test file I made locally with Gimp works but also already worked with the existing code because the EXIF data is after the area that got searched.

In T27707#4921804, @brion wrote:

In T27707#4920735, @AlexisJazz wrote:

Though this is not needed to remove this MIME sniffing check (or at the very least, make it optional), as I've shown it can't be exploited unless the webserver is misconfigured. So it doesn't apply to Wikimedia projects.

Can you clarify what you mean by "misconfigured"? No special configuration was required with exploit PNGs when I first tested this in 2005. ISTR that JPEG and GIF were not vulnerable when served with their native Content-Types because they have a different priority than PNG in the type checking.
The original 'naughty.png' sample from my 2005 testing is now available at https://brionv.com/misc/naughty.png -- note the way I embedded the HTML using a custom ancillary chunk might no longer be considered legit as some apps reject it now, but it could likely be retooled using a standard comment chunk. Breakdown of it is like this:

IE 8 on Windows 7 doesn't seem to be vulnerable and displays the image without running the script, but I still don't have an IE 6 or 7 test environment handy.

In T27707#4921089, @Aklapper wrote:

Logins should definitely be disabled for those.

This feels off-topic for this task. See https://wikitech.wikimedia.org/wiki/HTTPS/Browser_Recommendations#For_Users_of_Microsoft_Windows etc.

In T27707#4918648, @Tgr wrote:

In T27707#4915179, @brion wrote:

• This takes IE 6 off the table entirely.

IIRC we still get a bunch of IE6/7 visits from places which man-in-the-middle SSL connections.

Logins should definitely be disabled for those. That man (the one in the middle) can collect all their passwords. And to simplify things, those 4 users per million using IE7 on Vista should also have logins disabled. So logins can be disabled for all IE6 and IE7 users.

In T27707#4915140, @brion wrote:

The chance that someone would visit using an old IE version was probably 50% when the code was originally added; IE had a very high marketshare in the early 2000s. However at this point we can't even be accessed in IE 6 as far as I know (due to servers dropping old TLS versions for HTTPS). I think it's pretty fair to change the balance of what we check for.

In T27707#4918046, @Bawolff wrote:

In T27707#4918032, @AlexisJazz wrote:

In T27707#4918013, @Bawolff wrote:

As the file extension is restricted to a whitelist, the MIME type beyond the control of the attacker and invalid file signatures are also rejected, we must.. find a way around that I guess?

Usually in this scenario, its assumed file ext is .jpg, and the normal unix mime detection stuff would detect it as JPEG. But IE7 would in theory still treat as html (I think).

Actually no, the article Ghouston linked explains:

With the common GIF, JPEG and PNG formats, the browser ignores the result of MIME sniffing, as long as the filename extension, Content-Type and signature, all indicate the same type. Only if the results are inconsistent will Internet Explorer handle the file as the type identified by MIME sniffing.

So we really need to follow the Zany Scheme to exploit this. The scheme is a joke, but not inaccurate.

I think the microsoft docs say something different then Ghouston does:

If the server-provided MIME type is either known or ambiguous, the buffer is scanned in an attempt to verify or obtain a MIME type from the actual content. If a positive match is found (one of the hard-coded tests succeeded), this MIME type is immediately returned as the final determination, overriding the server-provided MIME type (this type of behavior is necessary to identify a .gif file being sent as text/html). During scanning, it is determined if the buffer is predominantly text or binary. Note In Microsoft Internet Explorer 6 for Windows XP Service Pack 2 (SP2), the MIME type "text/plain" is not ambiguous, and is never rendered as HTML in the restricted zone, even if the content suggests that this is the correct format.

From https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/ms775147(v=vs.85)
Otherwise, this would basically be a non-issue even at the time [Additional cause for concern is I think IE6 can sometimes take extensions from query parameter part of the url]

In T27707#4918013, @Bawolff wrote:

As the file extension is restricted to a whitelist, the MIME type beyond the control of the attacker and invalid file signatures are also rejected, we must.. find a way around that I guess?

Usually in this scenario, its assumed file ext is .jpg, and the normal unix mime detection stuff would detect it as JPEG. But IE7 would in theory still treat as html (I think).

In T27707#4915179, @brion wrote:

Double-checking:

• IE 6 and 7 were both available on Windows XP, which includes TLS 1.0 support but not TLS 1.1 or 1.2. However IE 6-8 on XP fail to work anyway due to lack of SNI.
  • This takes IE 6 off the table entirely.
• IE 7 on Vista should still work correctly with TLS 1.0, 1.1, or 1.2 -- https://blogs.msdn.microsoft.com/kaushal/2011/10/02/support-for-ssltls-protocols-on-windows/
• IE8 supports X-Content-Type-Options: nosniff header; if we don't already use it consistently, applying this on all file views would resolve all sniffing issues on IE 8

So the intersection of the danger zone, for Wikimedia sites, is:

• IE 7 running on Windows Vista, user logged in

(This would be a bad experience for the user with no JS and probably no or broken styles, and it's going to be a very small percentage of people that will hit it ever.)
For small internal sites, I assume you're just not going to have IE 6/7 so it shouldn't be a problem. ;)
I think it's reasonably fair to just drop the anti-sniffing checks at this point, though we might want to consider explicitly disallowing logins on IE 7.

In T214853#4914769, @Zoranzoki21 wrote:

I tested, works as expected. And in browser and via command line. Can you still reproduce this?

zoranzoki21@zoranzoki21-eMachines-E525:~$ curl -I https://commons.wikimedia.org/wiki/Special:Log/GifTagger
HTTP/2 200 
date: Mon, 28 Jan 2019 16:25:15 GMT
content-type: text/html; charset=UTF-8
server: mw1325.eqiad.wmnet
x-content-type-options: nosniff
p3p: CP="This is not a P3P policy! See https://commons.wikimedia.org/wiki/Special:CentralAutoLogin/P3P for more info."
x-powered-by: HHVM/3.18.6-dev
content-language: en
expires: Thu, 01 Jan 1970 00:00:00 GMT
x-frame-options: DENY
backend-timing: D=284163 t=1548692715609284
vary: Accept-Encoding,Cookie,Authorization,X-Seven
content-encoding: gzip
x-varnish: 500400808, 245907591, 227728396
via: 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1)
accept-ranges: bytes
age: 0
x-cache: cp1075 pass, cp3032 pass, cp3040 pass
x-cache-status: pass
server-timing: cache;desc="pass"
strict-transport-security: max-age=106384710; includeSubDomains; preload
set-cookie: WMF-Last-Access=28-Jan-2019;Path=/;HttpOnly;secure;Expires=Fri, 01 Mar 2019 12:00:00 GMT
x-analytics: ns=-1;special=Log;https=1;nocookies=1
x-client-ip: 109.245.111.9
cache-control: private, s-maxage=0, max-age=0, must-revalidate
set-cookie: GeoIP=RS:00:Belgrade:44.82:20.47:v4; Path=/; secure; Domain=.wikimedia.org

In T27707#4910850, @Ghouston wrote:

Editing Exif without potentially breaking something is difficult. A lot of extensions have been added to JPEG over the years, and some of them contain pointers to data outside their own segments. Things like maker notes (proprietary and not fully documented) and Multi-Picture Format (which allows putting multiple images in a single JPEG file). They all need to be unpacked, adjusted, and repacked. The only practical method is probably to use ExifTool, and even that isn't perfect.

In T213885#4907782, @Jdforrester-WMF wrote:

In T213885#4907757, @AlexisJazz wrote:

@Jdforrester-WMF I don't really know this stuff, but could it be a problem that you are calling getMediaInfoViewRegex before you've defined it?

Sadly, no, that code file is loaded in one go before processing. The subsequent operations to name the slot clearly take place (otherwise it'd say <mw:mediainfoslotheader /> rather than "Structured data".

@Jdforrester-WMF I don't really know this stuff, but could it be a problem that you are calling getMediaInfoViewRegex before you've defined it?

In T214003#4897627, @Tgr wrote:

@AlexisJazz If you mean at the exact same time, it's extra work and does not seem necessary. If you mean within a few days of each other, that's doable, sure. I was waiting for a reply on my comment.

As zhuyifei1999 says above, only the order matters. If the group change happens before the UW change, people would see the "Share images from Flickr" button but will be stopped by the abuse filter. Those who decide to ignore the abuse filter (just hit "upload" again) would need their uploads to be manually fixed. Which is doable for a few days, probably, as I've dealt with it before for extended uploaders, but messy and no smooth user experience. Merge the UW change first and everything will be fine.

@Multichill where should I propose a change? It should be changed globally, so I guess somewhere on meta?

It would be ideal if https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/485487/ (Merge the "extended-uploader" and "autopatrolled" user groups on Commons) and https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/UploadWizard/+/485141/ (mw.FlickrChecker: Use {{flickrreview}}) could be merged at the same time. (or merge the UploadWizard patch first, that'll be fine too)