Unable to continue/finish uploading chunked file
Closed, InvalidPublic
Actions

Assigned To

Authored By

	XZise
	Sep 12 2015, 10:51 PM

Description

While working on https://gerrit.wikimedia.org/r/#/c/234851/ I'm unable to finish a chunked file. Interestingly there are two tests. One is uploading a file in chunked mode which ignores warnings (as the target file already exist and there are duplicates). Another is uploading a file in chunked mode too but it doesn't ignore warnings. From that I get an offset and filekey and then it manually continues the upload in chunked mode again. But then it complains with stashfailed: No chunked upload session with this key.

Now it might be that pywikibot is screwing this up, but I logged the test and the request it send to the server mentions a filekey which I can use with query+stashimageinfo.

Related Objects

Mentioned In: T151562: ApiError on chunked upload with callable ignore_warnings
T133288: Fix test for TestUpload and remove expectedFailure decorator
rPWBCace3ca9c53a1: [FEAT] upload: Get stash info before continuing
Mentioned Here: T112405: First chunk can be smaller than 1024 bytes

Event Timeline

XZise created this task.Sep 12 2015, 10:51 PM

XZise raised the priority of this task from to Needs Triage.

XZise updated the task description. (Show Details)

XZise added projects: Pywikibot, MediaWiki-Action-API, MediaWiki-Uploading.

XZise subscribed.

Restricted Application added a project: Multimedia. · View Herald TranscriptSep 12 2015, 10:51 PM

Restricted Application added subscribers: pywikibot-bugs-list, Steinsplitter, Aklapper. · View Herald Transcript

XZise mentioned this in rPWBCace3ca9c53a1: [FEAT] upload: Get stash info before continuing.Sep 12 2015, 11:03 PM

Can you provide the actual requests, or a set of pywikibot or (ideally) curl commands reproducing the issue?

Restricted Application added a subscriber: Matanya. · View Herald TranscriptSep 13 2015, 1:04 AM

The failing test test_continue_filekey_chunked is in upload_tests

https://github.com/wikimedia/pywikibot-core/commit/409e592ab2d70663a82e777743ce7687b3f34065#diff-5a70e7141475f090fe760ee4c0e5c7b3R121

Instructions on how to run specific tests can be found at https://github.com/wikimedia/pywikibot-core/tree/master/tests#specific-tests

(This is shamelessly copied from T112405#1635010 so logging in and getting the token works the same)

Okay here are the curl commands to reproduce it. If you are logged in you can skip the two block. If you have already an CSRF token you can also skip the third block. The login will store the cookie into the cookie.lwp file.

Login

$ PASS=
$ NAME=
$ curl -b cookie.lwp -c cookie.lwp -d action=login -d lgname=$NAME -d lgpassword=$PASS -d format=json https://test.wikipedia.org/w/api.php
{"login":{"result":"NeedToken","token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","cookieprefix":"testwiki","sessionid":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}}

Now use the token in the next request and you are logged in:

$ curl -b cookie.lwp -c cookie.lwp -d action=login -d lgname=$NAME -d lgpassword=$PASS -d lgtoken=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -d format=json https://test.wikipedia.org/w/api.php
{"login":{"result":"Success","lguserid":x,"lgusername":"x","lgtoken":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","cookieprefix":"testwiki","sessionid":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}}

CSRF token

Now request the CSRF token (I request the userinfo too, to verify that I'm logged in):

$ curl -b cookie.lwp -d action=query -d meta="userinfo|tokens" -d type=csrf -d format=json https://test.wikipedia.org/w/api.php 
{"batchcomplete":"","query":{"userinfo":{"id":x,"name":"x"},"tokens":
{"csrftoken":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+\\"}}}
$ TOKEN="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+\\"

Upload

So now we get to this bug. I'm using the MP_sounds.png test file and split it into two chunks:

$ wget https://raw.githubusercontent.com/wikimedia/pywikibot-core/master/tests/data/images/MP_sounds.png
$ split -a 1 --numeric-suffixes=1 -b 1024 MP_sounds.png c1024.
$ curl -b cookie.lwp -F format=json -F action=upload -F stash=1 -F filesize=1276 -F filename=File:MP_sounds-pwb.png -F token=$TOKEN -F offset=0 -F chunk=@c1024.1 https://test.wikipedia.org/w/api.php
{"upload":{"result":"Warning","warnings":{"badfilename":"MP_sounds-pwb.png","exists":"MP_sounds-pwb.png"},"filekey":"13fzjy4zqofo.ty2ufl.28718.png","sessionkey":"13fzjy4zqofo.ty2ufl.28718.png"}}
$ FILEKEY=13fzjy4zqofo.ty2ufl.28718.png

It is important to store the filekey. We can also verify that something was actually uploaded using query+stashimageinfo:

$ curl -b cookie.lwp -d format=json -d action=query -d prop=stashimageinfo -d siifilekey=$FILEKEY -d siiprop="size|sha1|timestamp" https://test.wikipedia.org/w/api.php
{"batchcomplete":"","query":{"stashimageinfo":[{"timestamp":"2015-09-13T10:06:46Z","size":1024,"width":24,"height":27,"sha1":"3503db342c8dfb0a38db0682b7370ddd271fa163"}]}}
$ sha1sum c1024.*
3503db342c8dfb0a38db0682b7370ddd271fa163  c1024.1
2ed6566594d8893d55f5fbbc279f096ee157e28d  c1024.2

So the uploaded chunk seems fine, now continue uploading the second chunk:

$ curl -b cookie.lwp -F format=json -F action=upload -F stash=1 -F filesize=1276 -F filename=File:MP_sounds-pwb.png -F token=$TOKEN -F offset=1024 -F filekey=$FILEKEY -F chunk=@c1024.2 -F ignorewarnings=1 https://test.wikipedia.org/w/api.php
{"servedby":"mw1017","error":{"code":"stashfailed","info":"No chunked upload session with this key","*":"See https://test.wikipedia.org/w/api.php for API usage"}}

And tada the error using CURL so pywikibot is not actually failing here.

matmarex added subscribers: matmarex, Anomie.Sep 13 2015, 6:11 PM