Any idea how it can be validated that the upload was successfull even if the status response was 504?

@Inductiveload : You may decrease the retries count by changeing max_retries config variable. This is also possible as command line argument, either if your script uses handle_args() function or with the pwb.py wrapper script like

pwb.py -max_retries:2 <yourscript> [<youroptions>]

but pwb.py is not shipped with Pywikibot pypi package (yet)

Does it help?

It helped a bit, though I have to handle errors very carefully around such calls.

I know the overall "fault" is at the MediaWiki end for returning a 504 for this at all. Perhaps there's a way to catch the 504 and "poll" for an existing file? Or maybe that should just be done in user code?

The problem is the 504 timeout code which comes prior other warnings like

upload_warnings = {
    # map API warning codes to user error messages
    # {msg} will be replaced by message string from API response
    'duplicate-archive':
        'The file is a duplicate of a deleted file {msg}.',
    'was-deleted': 'The file {msg} was previously deleted.',
    'emptyfile': 'File {msg} is empty.',
    'exists': 'File {msg} already exists.',
    'duplicate': 'Uploaded file is a duplicate of {msg}.',
    'badfilename': 'Target filename is invalid.',
    'filetype-unwanted-type': 'File {msg} type is unwanted type.',
    'exists-normalized': 'File exists with different extension as '
                         '"{msg}".',
    'bad-prefix': 'Target filename has a bad prefix {msg}.',
    'page-exists':
        'Target filename exists but with a different file {msg}.',

    # API-returned message string will be timestamps, not much use here
    'nochange': 'The upload is an exact duplicate of the current '
                'version of this file.',
    'duplicateversions': 'The upload is an exact duplicate of older '
                         'version(s) of this file.',
}

And retrying is an usual behaviour with this code. But maybe we could check for the existing file after it.

I notice that the Internet-Archive project tag was automatically added to this task. Does this bug happen only on archive.org links, or does it happen in other cases as well?

@Harej Indeed it is not exclusive to the IA, it's any file which times out on upload. The IA is, however, an excellent source of files that do that! T292954 is making progress but it would be good to not have such a code path if avoidable.