This error seems to be constantly reproducible for this specific file (none of the upload attempts succeeded), but it was not previously encountered for other files.

I downloaded it from https://v2c.wmflabs.org/temp.webm and the file is 125164628 bytes for me, not 125164627.

$ du --bytes temp.webm
125164628       temp.webm

$ sha1sum temp.webm
e25042092c55f6ab12b178323e75d471cbc009c5 *temp.webm

Pywikibot seems to also think the file is 125164628 bytes (u'filesize': [125164628]), but it can't read/upload the last byte? First it sends a chunk one byte smaller than it claims (resulting in 'Unexpected offset' log warning from Pywikibot), then tries to send the last byte again but sends zero bytes (resulting in 'chunk-too-small' error from the API).

I don't have Pywikibot set up, but I was able to upload the file to stash to Commons with UploadWizard.

Indeed:

01:45:56 0 ✓ zhuyifei1999@encoding01: ~$ ls -al /srv/v2c/ssu/temp.webm
-rw-r--r-- 1 root root 125164628 Apr 14 12:10 /srv/v2c/ssu/temp.webm

I guess that was my error copy-pasting.

The interesting thing is, the last byte is a \r character:

$ cat /srv/v2c/ssu/temp.webm | xxd -b | tail
775dc1c: 10001101 11110111 10000001 00000001 11110001 10000100  ......
775dc22: 00000110 00000110 01111011 00111111 11110000 10000010  ..{?..
775dc28: 00000010 00100011 10111011 10010011 10110011 10000010  .#....
775dc2e: 11011000 11100101 10110111 10001101 11110111 10000001  ......
775dc34: 00000001 11110001 10000100 00000110 10001011 00010100  ......
775dc3a: 11011111 11110000 10000010 00000010 00001110 10111011  ......
775dc40: 10010011 10110011 10000010 11100111 10100000 10110111  ......
775dc46: 10001101 11110111 10000001 00000001 11110001 10000100  ......
775dc4c: 00000110 11111111 11011000 11101101 11110000 10000010  ......
775dc52: 00000010 00001101

I guess this is a pywikibot bug (or a python bug).

It could potentially be a WMF severs' misconfiguration, too.

The log you pasted above seems to have it handled correctly:

Content-disposition: form-data; name="chunk"; filename="FAKE-NAME"\nContent-Transfer-Encoding: binary\n\n\r\n

Note the '\r'.

Oooh, wait. I'm pretty sure that according to the spec, the request body should be using \r\n (CRLF) for line breaks, not just \n (LF). I guess the server is permissive and accepts it anyway, and in this case it can't tell that the '\r' wasn't meant to be part of the line break.

http://stackoverflow.com/questions/3086860/how-do-i-generate-a-multipart-mime-message-with-correct-crlf-in-python

I thought of some ways to fix/workaround this issue:

• WONTFIX for python 2 and move on to python 3 - really? Tool labs still have python 2 by default.
• Use the hack mentioned in StackOverflow (replace every \n with \r\n) - since we are working with binary data, this isn't desirable either.
• Code our own MIMEMultipart - I'm not an expert on how http transfers file data, so idk how difficult this would be.
• Base64 encode file contents - This would mean an increase in the bytes that are transferred for each upload, so users might feel a longer upload time.

Base64 encode file contents - This would mean an increase in the bytes that are transferred for each upload, so users might feel a longer upload time.

Since this is only a problem for files/chunks ending with '\r', I think it'd be a reasonable workaround if applied only in this case.

@Anomie is base64 encode supported by api action=upload?

Change 636164 had a related patch set uploaded (by Mpaa; owner: Mpaa):
[pywikibot/core@master] [bugfix] upload a file that ends with the '\r' byte

https://gerrit.wikimedia.org/r/636164

In T132676#6576561, @gerritbot wrote:

Change 636164 had a related patch set uploaded (by Mpaa; owner: Mpaa):
[pywikibot/core@master] [bugfix] upload a file that ends with the '\r' byte

https://gerrit.wikimedia.org/r/636164

Added to my site.py, but did not seem to stop the bug. Not clear to me why, perhaps initial conditions not met. Instead my work-around of removing the last byte was still triggered by receiving the error.

If you want examples to test out, there are over 200 PDFs now listed at https://commons.wikimedia.org/wiki/Category:Work_around_of_byte_trimmed_for_chunk-too-small_API_error where the original PDFs are not uploaded. The target could be the same files, just try to overwrite with the true original PDFs.

@Fae, strange, it worked for me, I tried this file and the byte was added.
https://commons.wikimedia.org/w/index.php?title=File%3ABritish_Transport_Commission_Act_1949_%28UKLA_1949-29%29.pdf&type=revision&diff=500900036&oldid=500838588

site = pywikibot.Site('commons', 'commons')
page = pywikibot.FilePage(site, 'File:British_Transport_Commission_Act_1949_(UKLA_1949-29).pdf')
site.loadimageinfo(page, history=True)

{'timestamp': '2020-10-25T13:52:37Z',  'user': 'Mpaa', 'size': 4873007,  'width': 1239,  'height': 1754,  'pagecount': 44, ...
{'timestamp': '2020-10-25T13:00:08Z',  'user': 'Fæ',  'size': 4873006,  'width': 1239,  'height': 1754,  'pagecount': 44, ...

Change 636164 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] upload a file that ends with the '\r' byte

https://gerrit.wikimedia.org/r/636164

In T132676#6575067, @Mpaa wrote:

@Anomie is base64 encode supported by api action=upload?

Not in itself, but you can use MIME's Content-Transfer-Encoding to encode the data as base64 rather than binary.

Since I no longer work for the WMF, I'm not inclined to do much to do their work for them at this time.

In T132676#6577051, @Anomie wrote:

In T132676#6575067, @Mpaa wrote:

@Anomie is base64 encode supported by api action=upload?

Not in itself, but you can use MIME's Content-Transfer-Encoding to encode the data as base64 rather than binary.

Since I no longer work for the WMF, I'm not inclined to do much to do their work for them at this time.

OK, thanks.
As info for future readers, I tried but if I set base64 there is a mismatch with filesize. At the end it gives an error saying that filesize does not match (as if it is adding the size of the base64-encoded chunks, maybe, and then it gets more data than declared in filesize).

Looks like I have a related problem: my uploaded PDFs lost the trailing 0x0d (that is \r). Uploaded them with Pywikibot 10.7.4

For my uploaded PDFs, Commons file size is exactly local size minus 1 byte.

Commons SHA-1 equals SHA-1 of my local file with the last byte removed.

The missing last byte is always 0x0d (\r).

Example:

Kavkaz_1909_N47.pdf

• local size: 29165766
• commons size: 29165765
• local sha1: 0535a19f2894f826781d3a1372bddb8e88650f57
• commons sha1: 98ee9db1d5fe1072114d644d7e9d729690d014ca
• sha1(local[:-1]): 98ee9db1d5fe1072114d644d7e9d729690d014ca