Page MenuHomePhabricator

pywikibot.exceptions.FatalServerError due to \n in User-Agent value
Closed, ResolvedPublic

Description

Hello pywikibot maintainers,
My bot, that is using nightly version (/data/project/shared/pywikibot/core/ on toolforge), is stuck with the following error since April 20.
The zone of the codebase that corresponds was edited by @Xqt in the following commit, but I don't see immediately where is the mistake here: https://github.com/wikimedia/pywikibot/commit/77ecbd1a753a2b518d5be89c8627265caae92564
I guess backline char (\n) has no place in user-agent string.

tools.framabot@shell:~$ ~/venv3.11/bin/python3 /data/project/shared/pywikibot/core/pwb.py login
fatal: detected dubious ownership in repository at '/data/project/pywikibot/public_html/core'
To add an exception for this directory, call:

	git config --global --add safe.directory /data/project/pywikibot/public_html/core
ERROR: Traceback (most recent call last):
  File "/data/project/pywikibot/public_html/core/pywikibot/data/api/_requests.py", line 684, in _http_request
    response = http.request(self.site, uri=uri,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/comms/http.py", line 283, in request
    r = fetch(baseuri, headers=headers, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/comms/http.py", line 457, in fetch
    callback(response)
  File "/data/project/pywikibot/public_html/core/pywikibot/comms/http.py", line 337, in error_handling_callback
    raise FatalServerError(str(response))
pywikibot.exceptions.FatalServerError: Invalid leading whitespace, reserved character(s), or returncharacter(s) in header value: 'login (wikipedia:fr; User:Framabot) Pywikibot/9.2.0.dev1 (1\n) requests/2.31.0 Python/3.11.2.final.0'

Traceback (most recent call last):
  File "/data/project/shared/pywikibot/core/pwb.py", line 40, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/project/shared/pywikibot/core/pwb.py", line 36, in main
    runpy.run_path(str(path), run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/data/project/shared/pywikibot/core/pywikibot/scripts/wrapper.py", line 521, in <module>
    main()
  File "/data/project/shared/pywikibot/core/pywikibot/scripts/wrapper.py", line 505, in main
    if not execute():
           ^^^^^^^^^
  File "/data/project/shared/pywikibot/core/pywikibot/scripts/wrapper.py", line 492, in execute
    run_python_file(filename, script_args, module)
  File "/data/project/shared/pywikibot/core/pywikibot/scripts/wrapper.py", line 149, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "/data/project/shared/pywikibot/core/pywikibot/scripts/login.py", line 178, in <module>
    main()
  File "/data/project/shared/pywikibot/core/pywikibot/scripts/login.py", line 141, in main
    for arg in pywikibot.handle_args(args):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/bot.py", line 832, in handle_args
    pywikibot.Site()
  File "/data/project/pywikibot/public_html/core/pywikibot/__init__.py", line 243, in Site
    _sites[key] = interface(code=code, fam=fam, user=user)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/site/_apisite.py", line 140, in __init__
    self.login(cookie_only=True)
  File "/data/project/pywikibot/public_html/core/pywikibot/site/_apisite.py", line 400, in login
    if self.userinfo['name'] == self.user():
       ^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/site/_apisite.py", line 670, in userinfo
    uidata = uirequest.submit()
             ^^^^^^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/data/api/_requests.py", line 993, in submit
    response, use_get = self._http_request(use_get, uri, body, headers,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/data/api/_requests.py", line 684, in _http_request
    response = http.request(self.site, uri=uri,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/comms/http.py", line 283, in request
    r = fetch(baseuri, headers=headers, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/pywikibot/public_html/core/pywikibot/comms/http.py", line 457, in fetch
    callback(response)
  File "/data/project/pywikibot/public_html/core/pywikibot/comms/http.py", line 337, in error_handling_callback
    raise FatalServerError(str(response))
pywikibot.exceptions.FatalServerError: Invalid leading whitespace, reserved character(s), or returncharacter(s) in header value: 'login (wikipedia:fr; User:Framabot) Pywikibot/9.2.0.dev1 (1\n) requests/2.31.0 Python/3.11.2.final.0'
CRITICAL: Exiting due to uncaught exception FatalServerError: Invalid leading whitespace, reserved character(s), or returncharacter(s) in header value: 'login (wikipedia:fr; User:Framabot) Pywikibot/9.2.0.dev1 (1\n) requests/2.31.0 Python/3.11.2.final.0'
tools.framabot@shell:~$

Event Timeline

if somebody has the same issue right now, just append this value in your ~/.pywikibot/user-config.py (and adapt the values, for sure):
user_agent_format="MY_BOT_NAME using pywikibot on MY_WIKI_NAME"

It is also possible some stuff was done on the toolforge side and impacted the output of the version.
Anyway, a .strip() could be added somewhere to remove any unattended \n in versions around here in pywikibot/version.py.

Hi there.

I adjusted the nightly version file path in rPWBC77ecbd1a7 that Pywikibot could find the version information from https://pywikibot.org nightly dumps. Unfortunately there were old bugs introduced with rPWBC7446058 in line 321 and 326 within the getversion_nightly() function (somewhat Pywikibot 2.0rc1): the path was bent and the the strip() methods were removed. Therefore the getversion() function falls back to the package version which gives something like that:

{'tag': 'pywikibot/__init__.py', 'rev': '-1 (unknown)', 'date': '2024/04/28, 14:23:49', 'hsh': ''}

@zhuyifei1999 made a patch for this issue. I'll try to add some tests soon.

Xqt triaged this task as High priority.May 2 2024, 8:35 AM

Change #1026462 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [tests] Add version_tests.py

https://gerrit.wikimedia.org/r/1026462

Change #1026185 had a related patch set uploaded (by Zhuyifei1999; author: Zhuyifei1999):

[pywikibot/core@master] version.py: Remove line endings in getversion_nightly

https://gerrit.wikimedia.org/r/1026185

Change #1026462 merged by jenkins-bot:

[pywikibot/core@master] [tests] Add version_tests.py

https://gerrit.wikimedia.org/r/1026462

Change #1026185 merged by jenkins-bot:

[pywikibot/core@master] version.py: Remove line endings in getversion_nightly

https://gerrit.wikimedia.org/r/1026185

Change #1026634 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [tests] update version_tests.py

https://gerrit.wikimedia.org/r/1026634

Change #1026634 merged by jenkins-bot:

[pywikibot/core@master] [tests] update version_tests.py

https://gerrit.wikimedia.org/r/1026634

@Xqt @zhuyifei1999 thanks both of you for this really quick patch!