Presumably the same happens with 'raise httplib2.RedirectLimit' in the same method.

I think that instead of the current

try:
    (response, content) = httplib2.Http.request(
        self, uri, method, body, headers,
        max_redirects, connection_type
    )
except Exception as e:  # what types?
    # return exception instance to be retrieved by the calling thread
    return e

wrapper, we should actually just wrap all of def request with a try-catch block which returns e on error. Does that sound like the right solution to you?

I was thinking it should be in HttpProcessor.run , which has a try..finally , .. but not catch. depending on how bad the exception is (and it is best to assume the worst, like some state has been irreparably altered: https://gerrit.wikimedia.org/r/#/c/178789/ ), it would be sensible to replace the dead http worker with a new one.

Yes, that sounds reasonable. This does have some implications for when we can and cannot re-use a connection, though -- we should prevent spinning up a new connection for typical issues (e.g. 503s).

ya, we have a try.. catch inside .request , and it returns the exception to be processed by the client. I think that should not be changed, which is where 503's normally happen. It would only be unexpected exceptions which would be handled by HttpProcessor.run and result in a new worker being created.

I am also interested in why we need the special follow code. Im pretty sure it is being used because of %20 -> _ on MediaWiki sites, but that should only be needed for full urls to wiki pages, which is very rare occurrence in pywikibot core these days.