Page MenuHomePhabricator
Create Task
Maniphest T232672

API does not strip bidi characters (or trim whitespace) when validating IPs for 'user'-type parameters
Closed, ResolvedPublic
Actions

Description

Examples:

versus

Original report:

This started to happen before few days, but no for all IP addresses.

kizule@kizule:~/development/pywikibot-core$ python3 pwb.py patrol -usercontribs:‎'213.149.159.237' -ask
/home/kizule/development/pywikibot-core/pywikibot/config2.py:1138: _ConfigurationDeprecationWarning: "sysopnames" present in our user-config.py is no longer a supported configuration variable. Please inform the maintainers if you depend on it.
  'depend on it.'.format(_key), _ConfigurationDeprecationWarning)
Processing user: ‎213.149.159.237
Newpages:
Loading Корисник:Zoranzoki21/patrol_whitelist
WARNING: API error baduser_rcuser: Invalid value "‎213.149.159.237" for user parameter "rcuser".
Traceback (most recent call last):
  File "pwb.py", line 321, in <module>
    if not main():
  File "pwb.py", line 316, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "pwb.py", line 101, in run_python_file
    main_mod.__dict__)
  File "./scripts/patrol.py", line 505, in <module>
    main()
  File "./scripts/patrol.py", line 490, in main
    bot.run(feed)
  File "./scripts/patrol.py", line 286, in run
    for page in feed:
  File "./scripts/patrol.py", line 410, in api_feed_repeater
    for page in generator:
  File "/home/kizule/development/pywikibot-core/pywikibot/site.py", line 6627, in newpages
    for pageitem in gen:
  File "/home/kizule/development/pywikibot-core/pywikibot/data/api.py", line 2808, in __iter__
    self.data = self.request.submit()
  File "/home/kizule/development/pywikibot-core/pywikibot/data/api.py", line 2080, in submit
    raise APIError(**result['error'])
pywikibot.data.api.APIError: baduser_rcuser: Invalid value "‎213.149.159.237" for user parameter "rcuser". [help:See https://sr.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce&gt; for notice of API deprecations and breaking changes.]
CRITICAL: Exiting due to uncaught exception <class 'pywikibot.data.api.APIError'>
kizule@kizule:~/development/pywikibot-core$

Version of pywikibot:

kizule@kizule:~/development/pywikibot-core$ python3 pwb.py version
/home/kizule/development/pywikibot-core/pywikibot/config2.py:1138: _ConfigurationDeprecationWarning: "sysopnames" present in our user-config.py is no longer a supported configuration variable. Please inform the maintainers if you depend on it.
  'depend on it.'.format(_key), _ConfigurationDeprecationWarning)
Pywikibot: [ssh] pywikibot-core (e78bbf3, g11512, 2019/09/10, 11:30:54, ok)
Release version: 3.1.dev0
requests version: 2.21.0
  cacerts: /etc/ssl/certs/ca-certificates.crt
    certificate test: ok
Python: 3.7.3 (default, Aug 20 2019, 17:04:43) 
[GCC 8.3.0]
PYWIKIBOT_DIR: Not set
PYWIKIBOT_DIR_PWB: 
PYWIKIBOT_NO_USER_CONFIG: Not set
Config base dir: /home/kizule/development/pywikibot-core
Usernames for family "wikipedia":
	sr: Zoranzoki21 (also sysop)
kizule@kizule:~/development/pywikibot-core$

Details

SubjectRepoBranchLines +/-
API: Use ParamValidator librarymediawiki/coremaster+3 K -1 K
Customize query in gerrit

Related Objects

Event Timeline

Kizule created this task.Sep 11 2019, 8:33 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptSep 11 2019, 8:33 PM
Xqt added a project: MediaWiki-Action-API.Sep 12 2019, 10:55 AM
Restricted Application added a project: Platform Engineering. · View Herald TranscriptSep 12 2019, 10:55 AM
Anomie moved this task from Unsorted to Needs details or plan on the MediaWiki-Action-API board.EditedSep 12 2019, 1:15 PM
Anomie subscribed.
kizule@kizule:~/development/pywikibot-core$ python3 pwb.py patrol -usercontribs:‎'213.149.159.237' -ask

The value you're passing there is not just an IP address. It's an IP address prefixed with U+200E (LEFT-TO-RIGHT MARK), which makes it an invalid value.

The Unicode bidi override characters (U+200E, U+200F, and U+202A–U+202E) are stripped from actual usernames during canonicalization (as part of passing it through the page title parser), but the API parameter validation does not strip them from IP addresses when validating those. At a glance it looks like other code in MediaWiki that handles IP usernames specially also generally doesn't strip these characters.

The API validation code for IP usernames also doesn't trim leading or trailing whitespace, as is done for page titles.

I'm undecided whether this code should be updated to do that stripping, or if the non-stripping is acceptable behavior.

Anomie moved this task from Inbox to Triage Meeting Inbox on the Platform Engineering board.Sep 12 2019, 1:15 PM
Anomie renamed this task from API error baduser_rcuser in pywikibot (not for all IP addresses) to API does not strip bidi characters (or trim whitespace) when validating IPs for 'user'-type parameters.Sep 12 2019, 1:18 PM
Anomie updated the task description. (Show Details)Sep 12 2019, 1:21 PM
Anomie updated the task description. (Show Details)
Anomie updated the task description. (Show Details)
Kizule added a comment.Sep 12 2019, 1:45 PM
In T232672#5487675, @Anomie wrote:
kizule@kizule:~/development/pywikibot-core$ python3 pwb.py patrol -usercontribs:‎'213.149.159.237' -ask

The value you're passing there is not just an IP address. It's an IP address prefixed with U+200E (LEFT-TO-RIGHT MARK), which makes it an invalid value.

The Unicode bidi override characters (U+200E, U+200F, and U+202A–U+202E) are stripped from actual usernames during canonicalization (as part of passing it through the page title parser), but the API parameter validation does not strip them from IP addresses when validating those. At a glance it looks like other code in MediaWiki that handles IP usernames specially also generally doesn't strip these characters.

The API validation code for IP usernames also doesn't trim leading or trailing whitespace, as is done for page titles.

I'm undecided whether this code should be updated to do that stripping, or if the non-stripping is acceptable behavior.

I tried with 'IP address' and without it but same happening.

Xqt subscribed.Sep 12 2019, 4:09 PM

I tried with 'IP address' and without it but same happening.

Cannot follow. Can you explain your comment please.

Kizule added a comment.Sep 12 2019, 4:25 PM
In T232672#5488159, @Xqt wrote:

I tried with 'IP address' and without it but same happening.

Cannot follow. Can you explain your comment please.

I mean to I tried to use this formating: '127.0.0.1', "127.0.0.1" and 127.0.0.1

But same effect.

Kizule added a comment.Sep 12 2019, 4:26 PM

But thanks @Anomie for contributing to this task and I will when I come home (because I am in bus currently) to type IP address no copy-paste from srwiki recent changes.

Kizule triaged this task as Low priority.Sep 12 2019, 5:38 PM

Ok, I tried to type IP address, not to copy-paste and works.

kizule@kizule:~/development/pywikibot-core$ python3 pwb.py patrol -usercontribs:109.122.123.39 -ask
/home/kizule/development/pywikibot-core/pywikibot/config2.py:1138: _ConfigurationDeprecationWarning: "sysopnames" present in our user-config.py is no longer a supported configuration variable. Please inform the maintainers if you depend on it.
  'depend on it.'.format(_key), _ConfigurationDeprecationWarning)
Processing user: 109.122.123.39
Newpages:
Loading Корисник:Zoranzoki21/patrol_whitelist
Recentchanges:
User 109.122.123.39 has created or modified page Односи Србије и Шведске
Do you want to mark page as patrolled? ([y]es, [n]o, [q]uit): # test
Do you want to mark page as patrolled? ([y]es, [n]o, [q]uit): q

User quit PatrolBot bot run.
0/0 patrolled
kizule@kizule:~/development/pywikibot-core$ python3 pwb.py version
/home/kizule/development/pywikibot-core/pywikibot/config2.py:1138: _ConfigurationDeprecationWarning: "sysopnames" present in our user-config.py is no longer a supported configuration variable. Please inform the maintainers if you depend on it.
  'depend on it.'.format(_key), _ConfigurationDeprecationWarning)
Pywikibot: [ssh] pywikibot-core (cce0d30, g11513, 2019/09/12, 16:13:38, ok)
Release version: 3.1.dev0
requests version: 2.21.0
  cacerts: /etc/ssl/certs/ca-certificates.crt
    certificate test: ok
Python: 3.7.3 (default, Aug 20 2019, 17:04:43) 
[GCC 8.3.0]
PYWIKIBOT_DIR: Not set
PYWIKIBOT_DIR_PWB: 
PYWIKIBOT_NO_USER_CONFIG: Not set
Config base dir: /home/kizule/development/pywikibot-core
Usernames for family "wikipedia":
	sr: Zoranzoki21 (also sysop)
kizule@kizule:~/development/pywikibot-core$
Xqt removed a project: Pywikibot.Sep 13 2019, 4:55 AM

This is not pywikibot related because there where wrong chars in the command line and therefore this task looks invalid. Probably these invalid characters could be stripped during api call or the warning could be more informative. I don’t think that we should strip all command line arguments on pywikibot side but that would be an option too.

• eprodromou moved this task from Triage Meeting Inbox to Icebox on the Platform Engineering board.Sep 17 2019, 6:43 PM
Anomie moved this task from Needs details or plan to Needs Code on the MediaWiki-Action-API board.Sep 17 2019, 6:46 PM

So probably the most consistent thing to do here is to pass it through Title::newFromText( $value, NS_USER ) (then get the IP back out with ->getText()), like User::getCanonicalName() does for registered user names, before running the regexes to determine if it's an IP.

Anomie claimed this task.Dec 19 2019, 4:56 PM
Anomie moved this task from Needs Code to Needs Review on the MediaWiki-Action-API board.Jan 14 2020, 10:25 PM
gerritbot added a comment.Feb 4 2020, 8:39 PM

Change 434718 merged by jenkins-bot:
[mediawiki/core@master] API: Use ParamValidator library

https://gerrit.wikimedia.org/r/434718

Anomie closed this task as Resolved.Feb 4 2020, 8:41 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:43 PM
Novem_Linguae mentioned this in T313486: Daylight savings unit test failures on Microsoft Windows.Jul 21 2022, 10:14 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL