Page MenuHomePhabricator

upload.create_warnings_list() throws KeyError when ignore_warnings is true
Closed, ResolvedPublicBUG REPORT

Description

In current HEAD (PWB pre-7.0), imagetransfer.py triggers a Python KeyError in site/_apisite.py:upload.create_warnings_list() when ignore_warnings is true. The cause seems to be that create_warnings_list() is looking for an offset entry in the passed-in response dict which is (no longer?) being set for that code path instead of the _offset argument passed in through upload()s signature.

❯ python3 pwb.py imagetransfer -lang:commons -family:commons "File:filename.jpg" -tolang:en -tofamily:wikisource -force_if_shared
------------------------------------------------------------
0. Found image: [[File:filename.jpg]]
=={{int:filedesc}}==
…
============================================================

>>> Transfer [[commons:File:filename.jpg]] from commons:commons to wikisource:en

URL should be: https://upload.wikimedia.org/wikipedia/commons/b/b4/filename.jpg

The filename on the target wiki will default to: filename.jpg

Enter a better name, or press enter to accept: 
The suggested description is: …

Do you want to change this description? ([y]es, [N]o, [q]uit): 
Uploading file to wikisource:en...
ERROR: Upload error: 
Traceback (most recent call last):
  File "…pywikibot/specialbots/_upload.py", line 413, in upload_file
    success = imagepage.upload(file_url,
  File "…pywikibot/page/__init__.py", line 2456, in upload
    return self.site.upload(self, source_filename=filename, source_url=url,
  File "…pywikibot/site/_decorators.py", line 92, in callee
    return fn(self, *args, **kwargs)
  File "…pywikibot/site/_apisite.py", line 2968, in upload
    if ignore_warnings(create_warnings_list(result)):
  File "…pywikibot/site/_apisite.py", line 2595, in create_warnings_list
    return [
  File "…pywikibot/site/_apisite.py", line 2599, in <listcomp>
    _file_key, response['offset'])
KeyError: 'offset'
1 pages read
0 pages written
0 pages skipped
Execution time: 14 seconds
Read operation time: 14.0 seconds
Script terminated successfully.

The immediate fix is to use _offset instead of response['offset'] there, but these are not going to be identical, and it is possible this will cause problems in other code paths. From what I can tell from git blame, I conclude that this is the correct variable to use and any other code paths that break as a result should instead be fixed to update _offset. Someone that knows the code better will have to make that call though.

Details

Event Timeline

Xqt triaged this task as High priority.Feb 12 2022, 7:17 PM

Change 762101 had a related patch set uploaded (by Xover; author: Xover):

[pywikibot/core@master] pywikibot: Fix KeyError in create_warnings_list

https://gerrit.wikimedia.org/r/762101

This issue was introduced with rPWBC758de7a to solve T294916 (line 2882):

old code:

if not report_success:
    result.setdefault('offset', True)

new code:

if not report_success:
    if source_filename:
        offset = result.setdefault('offset', True)
    else:
        offset = False

Possibly we can remove offset and filekey when creating an UploadError due to T301633.

Xqt claimed this task.

Change 762101 merged by jenkins-bot:

[pywikibot/core@master] pywikibot: Fix KeyError in create_warnings_list

https://gerrit.wikimedia.org/r/762101