Page MenuHomePhabricator

Non transcluded variable in stashfailed error message
Closed, ResolvedPublic

Description

When trying to upload a specific .tif file through the API I get the following error message.

stashfailed: Cannot upload this file because Internet Explorer would detect it as "$1", which is a disallowed and potentially dangerous file type.

I'm not bothered by what caused it but I noticed the "$1" which looks like a un-transcluded variable in the message.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 8 2016, 12:45 PM

I'm unable to reproduce this locally. When I attempt to upload a file that triggers UploadBase::verifyMimeType() to return 'filetype-bad-ie-mime' as an error, it does not include the parsed output of the error message:

{
   "error" : {
      "code" : "verification-error",
      "info" : "This file did not pass file verification: Unknown error: \"filetype-bad-ie-mime\"",
      "details" : [
         "filetype-bad-ie-mime",
         "foo"
      ],
      "*" : "See http://localhost/w/api.php for API usage"
   }
}

Chunked upload gives a somewhat less useful message, but still not what's reported here:

{
   "error" : {
      "code" : "stashfailed",
      "info" : "filetype-bad-ie-mime",
      "*" : "See http://localhost/w/api.php for API usage"
   }
}

I did the upload through pywikibot but checked and the message did not originate from there.

@Lokal_Profil Can you upload the problematic file here to Phabricator? (Use the icon on the comment field toolbar.)

It would also be useful to know the exact version of pywikibot you're using, the exact command you're running, and the exact complete output.

API action=upload can return a "stashfailed" in approximately a billion different circumstances in one of its million different modes, and it's possible one of them handles the output incorrectly and outputs this (broken) message rather than an error code. But it's far easier for you to give us more details, so that we can tell which one you're running into, than for us to try to check them all.

It would also be useful to know the exact version of pywikibot you're using, the exact command you're running, and the exact complete output.

@Lokal_Profil: Could you provide this? ^

Appologies for the delay.

Pywikibot: I'm using the master/dev branch where I call the pywikibot.Site.upload() method.

I'll check if I still have the original error log (and the failing file) on the external drive provided by the GLAM. Note though that the file might be ~100MB.

Appologies for the delay.
Pywikibot: I'm using the master/dev branch where I call the pywikibot.Site.upload() method.
I'll check if I still have the original error log (and the failing file) on the external drive provided by the GLAM. Note though that the file might be ~100MB.

I've just checked the external disk and unfortunately I haven't kept the log files so I cannot give you a closer explanation of the source of the error. My hope was that it would be possible to grep for the string in the appropriate repo.

Anomie closed this task as Declined.Nov 8 2016, 3:58 PM

I've just checked the external disk and unfortunately I haven't kept the log files so I cannot give you a closer explanation of the source of the error. My hope was that it would be possible to grep for the string in the appropriate repo.

In that case I'm just going to close this since the grep didn't turn up any code paths that seem able to give the described behavior. Feel free to reopen if you manage to determine a method to reproduce it.

Lokal_Profil reopened this task as Open.EditedDec 8 2016, 8:09 PM

@Anomie Reopening this since it happened again and I have retained the file this time.
It is 30MB is there a preferred location to stick it?

In addition to the details I gave above (upoaded through pywikibot.Site.upload()) the other detail is that I did so in 5MB (5242880) chunks.

Anomie added a comment.Dec 8 2016, 8:18 PM

Note that rMW4e6810e4a2c1: API: i18n for warnings and errors (which should be deployed with 1.29.0-wmf.6) probably changed whatever code was giving you the issue.

My first thought would be to upload the file to Phab. It would also be helpful if you would post a full script that can be run to reproduce the error.

Lokal_Profil added a comment.EditedDec 8 2016, 8:52 PM

The following script is the minimal implementation (tested) which triggers the error (but requires pywikibot).

I could not upload the image to phab since it was to large but I've attached the associated info file (in case that somehow plays a role) and I stuck the image on dropbox.
Fotografi,_katedral_i_Salamanca_-_Hallwylska_museet_-_107303.tif
{F5013004}

1import pywikibot
2import codecs
3
4base_name = 'Fotografi,_katedral_i_Salamanca_-_Hallwylska_museet_-_107303'
5info_name = base_name + '.info'
6file_name = base_name + '.tif'
7chunk_size = 5*1048576
8
9site = pywikibot.Site('commons', 'commons')
10site.login()
11
12file_page = pywikibot.FilePage(site, file_name)
13info_file = codecs.open(info_name, 'r', 'utf-8')
14file_page.text = info_file.read()
15
16site.upload(file_page,
17 source_filename=file_name,
18 ignore_warnings=None,
19 report_success=False,
20 chunk_size=chunk_size)

Anomie added a comment.Dec 8 2016, 9:47 PM

Your script and the text of the dropbox link say "Fotografi,_katedral_i_Salamanca_-_Hallwylska_museet_-_107303", but the actual dropbox filename and the info file are "Chatelaine,_1700-tal_-_Hallwylska_museet_-_110501"?

Argh. Jupp that was a file with another upload problem.

Fotografi,_katedral_i_Salamanca_-_Hallwylska_museet_-_107303.tif and

are the right ones.

An update: Uploading the file via the api with chunked upload turned off does not trigger the error message.

The upload was accidental* but means any testing needs to be done Beta Commons instead (apologies for that).

*I'm trying to get some smaller .tif files mistakenly caught by the "This file contains HTML or script code that may be erroneously interpreted by a web browser." to upload by not chunking them.

I can confirm it now, but it's not an API bug.

To reproduce: Make a chunked upload where one of the chunks after the first fails the IE mime type test.

What happens here is that UploadFromChunks::verifyChunk() is getting the proper error indicator [ 'filetype-bad-ie-mime', 'application/x-msdownload' ] for one of the partial chunks and it's stupidly keeping only the 'filetype-bad-ie-mime' bit because of the incredibly weird way it uses UploadChunkVerificationException. It doesn't happen for non-chunked uploads because those don't even use UploadFromChunks, and if the failure happens for the first chunk (as in T147720#2705921) rather than a subsequent chunk it errors out via a different code path instead.

Restricted Application added a project: Multimedia. · View Herald TranscriptDec 9 2016, 4:55 PM

(The reason why the upload is failing at all is probably T143610. The only workaround is to try to upload in chunks of a different size and hope you don't hit it.)

Restricted Application added a subscriber: Matanya. · View Herald TranscriptDec 13 2016, 12:05 PM

For future reference (Dropbox links tend to die sooner or later), the problematic file is also available at https://commons.wikimedia.org/wiki/File:Fotografi,_katedral_i_Salamanca_-_Hallwylska_museet_-_107303.tif now.

Change 326944 had a related patch set uploaded (by Bartosz Dziewoński):
Do not lose message parameters in UploadFromChunks::verifyChunk()

https://gerrit.wikimedia.org/r/326944

Change 326944 merged by jenkins-bot:
Do not lose message parameters in UploadFromChunks::verifyChunk()

https://gerrit.wikimedia.org/r/326944

matmarex closed this task as Resolved.Dec 14 2016, 7:18 AM
matmarex claimed this task.
matmarex removed a project: Patch-For-Review.

This should fix the error message. Chunked-uploading the file will still be failing, even though it shouldn't be, that's T143610.