TestHttpStatus::test_follow_redirects fails with OAUTH credentials in user-config.py
Open, HighPublicBUG REPORT
Actions

Assigned To

Authored By

	RoySmith
	May 12 2023, 9:03 PM

Description

Steps to replicate the issue (include links if applicable):

checkout pywikibot/core.git to 77482f4bab1822c7b604f7d1a4787ad2742ec991
pytest tests/http_tests.py::TestHttpStatus::test_follow_redirects

What happens?:
Different results on different platforms.

On MacOS 12.6.1, Python 3.9.13:

E       AssertionError: '//en.wikipedia.org/wiki/Main_Page' not found in 'https://en.wikipedia.org/wiki/Main%20Page'

On debian 11.7 (bullseye), Python 3.9.2, the test passes.

On both platforms, running curl manually on the command line shows a two-step redirect:

http://en.wikipedia.org/wiki/Main%20Page -> https://en.wikipedia.org/wiki/Main%20Page
https://en.wikipedia.org/wiki/Main%20Page -> https://en.wikipedia.org/wiki/Main_Page

If I run this on MacOS:

r = requests.get('http://en.wikipedia.org/wiki/Main%20Page')
print(len(r.history))
print(r.url)

it prints what I would expect:

2
https://en.wikipedia.org/wiki/Main_Page

but single-stepping through the test code in the debugger, after:

response = session.request(method, uri,
                           headers=headers, auth=auth, timeout=timeout,
                           **kwargs)

in pywikibot/comms/http.py, I get:

len(response.history)
1
response.url
'https://en.wikipedia.org/wiki/Main%20Page'

Unclear at this point why it's not doing the 2nd redirect. I'll keep digging.

Related Objects

Mentioned In: T336624: MediaWiki does not redirect URL with OAuth credentials supplied
Mentioned Here: rPWBC77482f4bab18: Merge "[doc] Publish textlib._get_regexes()"

Event Timeline

RoySmith created this task.May 12 2023, 9:03 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMay 12 2023, 9:03 PM

RoySmith claimed this task.May 12 2023, 9:03 PM

I see what's going on. This is somehow related to authentication. I have OAUTH credentials in my user-config.py file. It's unclear why it should cause this behavior, but if I comment out the:

authenticate['en.wikipedia.org'] = (...)

line in user-config.py, the test passes.

RoySmith renamed this task from TestHttpStatus::test_follow_redirects fails on some platforms to TestHttpStatus::test_follow_redirects fails with OAUTH credentials in user-config.py.May 12 2023, 9:38 PM

Xqt triaged this task as High priority.May 13 2023, 2:43 PM

It's possible this is a bug in the upstream requests library. I can reproduce this (on both MacOS and Debian) with:

#!/usr/bin/env python

import requests
from requests_oauthlib import OAuth1

r1 = requests.get('https://en.wikipedia.org/wiki/Main%20Page', auth=OAuth1('', '', '', ''))
r2 = requests.get('https://en.wikipedia.org/wiki/Main%20Page', auth=None)

print(f"{r1.url=}")
print(f"{r2.url=}")

(requests) Roys-MBP [requests] /Users/roy/dev/requests/venv/bin/python /Users/roy/dev/requests/src/try.py
r1.url='https://en.wikipedia.org/wiki/Main%20Page'
r2.url='https://en.wikipedia.org/wiki/Main_Page'

Using my real OAuth credentials gives the same results as the dummy values shown above.

On the other hand, maybe this is a server-side issue? I'm all the way down to

resp = conn.urlopen(
    method=request.method,
    url=url,
    body=request.body,
    headers=request.headers,
    redirect=False,
    assert_same_host=False,
    preload_content=False,
    decode_content=False,
    retries=self.max_retries,
    timeout=timeout,
    chunked=chunked,
)

in requests/adapters.py. url is '/wiki/Main%20Page' and I'm getting back a 200 status.

RoySmith mentioned this in T336624: MediaWiki does not redirect URL with OAuth credentials supplied.May 13 2023, 6:36 PM

TestHttpStatus::test_follow_redirects fails with OAUTH credentials in user-config.pyOpen, HighPublicBUG REPORTActions

Description

Related Objects

Event Timeline

TestHttpStatus::test_follow_redirects fails with OAUTH credentials in user-config.py
Open, HighPublicBUG REPORT
Actions