Page MenuHomePhabricator

UploadBase::detectScript() is executed for partially uploaded files (verifyPartialFile()), not only complete ones (verifyFile()) which it expects, causing false positives
Open, LowPublic

Description

UploadBase::detectScript() is executed for partially uploaded files (verifyPartialFile()), not only complete ones (verifyFile()). However, it expects to only work with complete files (e.g. it checks the first two bytes to determine UTF-16 endianness, and some other checks).

This means that the checks meant to run only on first 1 KB of the file, or first 256 KB, etc., in fact run for the first x KB of every uploaded chunk, accidentally checking things in the middle of large files.

Here's an example file that falsely fails with 'uploadscripted' when every 5 MB chunk is checked, but passes when uploaded in one large chunk:

(~25 MB TIFF file, source). The easiest way to reproduce is to try uploading the file with UploadWizard, which uses 5 MB chunked uploads. The cause is the presence of the string '<PrE' at offset 10332876 within the file.

Even if an individual chunk of a file was nefarious, only the uploader is able to access stashed uploads (including chunked ones), so allowing them is not a security vulnerability.

Note that this is not a security issue – it can only cause false positives (file upload prevented for good files), never false negatives (file upload allowed for bad files). I am tagging Security since it's security-relevant code, and because I would like the explanation above and the patch carefully reviewed from the security perspective.

(Originally reported at https://commons.wikimedia.org/wiki/Commons:Upload_help#.22Internal_error:_Server_failed_to_store_temporary_file..22_.28uploadscripted.29.)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 306057 had a related patch set uploaded (by Bartosz Dziewoński):
UploadBase: Move detectScript() call from verifyPartialFile() to verifyFile()

https://gerrit.wikimedia.org/r/306057

matmarex triaged this task as Medium priority.Aug 22 2016, 9:53 PM

I'm not sure how confident I am that other users really can't view stash. In current WMF config, its possible to be able to view thumbnails of stashed files by other users, and I imagine third parties would be less locked down.

I'm not sure how confident I am that other users really can't view stash. In current WMF config, its possible to be able to view thumbnails of stashed files by other users, and I imagine third parties would be less locked down.

Right, but that's a bug, see T130436. Vanilla MediaWiki with all default settings (other than file upload enabled) correctly prevents viewing them (although it's possible to accidentally break this, see T144176).

matmarex added subtasks: Restricted Task, Restricted Task.Sep 4 2016, 8:03 AM

Change 306057 abandoned by Bartosz Dziewoński:
UploadBase: Move detectScript() call from verifyPartialFile() to verifyFile()

https://gerrit.wikimedia.org/r/306057

matmarex lowered the priority of this task from Medium to Low.
matmarex removed a project: Patch-For-Review.

Since the blocker tasks would require some work from WMF Operations before this change could go live on Wikimedia sites, and I have no power over them to make them prioritize it, this goes on the back burner. Frankly it's a low-severity issue and I can't justify to myself bugging people about it.

Lokal_Profil subscribed.

I have ~125 tif files on a disk which all fail due to this. The ones <100 MB I can re-upload un-chunked (which is a pain but works sooner or later) but anything bigger is completely blocked by this bug (different chunk sizes haven't solved it for me either although I could of course go on experimenting).

@matmarex can you add me to the subtasks so that I can read them ?

Change 998655 had a related patch set uploaded (by Maddog; author: Maddog):

[mediawiki/core@master] upload: Allow disabling of per-chunk file verification during chunked upload

https://gerrit.wikimedia.org/r/998655

I've just submitted a patch that (in my opinion) solves this issue in a pretty practical way: it provides a configuration parameter to allow a site to skip trying to apply any file verification to the individual chunks of a chunked upload. This way, a site admin can decide for themselves if the checks improve security on their site, and how that improvement weighs against the problems caused by the checks. (E.g., in my case, with a site that serves in part as a document archive, the false positives are simply unacceptable.)

Is this just about the checks for internet explorer 6? That seems so far past relavence that we should just remove them.

Is this just about the checks for internet explorer 6? ...

No --- there are still false positive rejections of uploads (due to detectScript(), etc) even with VerifyMimeTypeIE turned off. (Turning off the IE6 bits does reduce the incidence of false positives, in my experience, but does not eliminate them.)

The current verify-each-chunk behavior also results in attempting to extract metadata from each and every chunk. As elaborated in my commit message:

Attempting to verify chunks taken from the middle of files also causes
non-sensical behavior. When uploading a PDF file, the verification checks
call MediaHandler::getSizeAndMetadataWithFallback(), and for a PDF this
invokes executing an external program, pdfinfo, via a firejail sandbox.
In a chunked upload, pdfinfo is spawned for *every single chunk* and every
one of those calls fails anyway, because pdfinfo is being fed an incomplete
PDF file or random gibberish taken from the middle of a PDF file. Uploading
a 10MB PDF file will result in 10 pointless executions of pdfinfo (which
also requires copying the chunks into temporary sandbox directories).

Being able to simply turn off the verify-each-chunk behavior mostly eliminates that, too.