Page MenuHomePhabricator

APIErrors for maxlag parameter higher than 9 starting February 12
Closed, ResolvedPublic

Description

On T245209: Create symlink to PWB py2 version in /shared/pywikibot I said that several py2 scripts I was maintaining were blocked from the same time, few days ago, and that they share the particularity of using the regularly updated version of the library on the tools. So I deduced, obviously too quickly, that pwb had voluntarily pushed a py3 patch only. @Dvorapa tells me that this is not the case, so I report the problem by creating this ticket.

First and last occurrence seen on my logs on February 13 2020 on py2 script using /shared/pywikibot/core/pwb.py (up-to-date to last master version). Not seen after switching to /data/project/pywikibot/public_html/core_python2/pwb.py (python2 branch) the same day. Not part of bot code was edited to perform the switch successfully.

APIError: badinteger: Invalid value "3|0" for integer parameter "maxlag".
[help: See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.]
  File "twitterwpfr.py", line 43, in run
    self.imageDuJour()
  File "twitterwpfr.py", line 75, in imageDuJour
    assert pywikibot.FilePage(pywikibot.Site('commons', 'commons'), filename).exists()
  File "pywikibot/page.py", line 815, in exists
    return self.pageid > 0
  File "pywikibot/page.py", line 292, in pageid
    self.site.loadpageinfo(self)
  File "pywikibot/site.py", line 2987, in loadpageinfo
    self._update_page(page, query)
  File "pywikibot/site.py", line 2967, in _update_page
    for pageitem in query:
  File "pywikibot/data/api.py", line 3001, in __iter__
    for result in super(PropertyGenerator, self).__iter__():
  File "pywikibot/data/api.py", line 2825, in __iter__
    self.data = self.request.submit()
  File "pywikibot/data/api.py", line 2094, in submit
    raise APIError(**result['error'])

On another bot I maintain, using py2 and same pwb.py endpoint, with username that contains non ascii letters. 792 exceptions starting February 12 2020 07:01:50 CET. Logged version() output at this time was [https] r-pywikibot-core.git (3dac747, g1, 2020/02/11, 17:23:13, ok).

OtherPageSaveError: Edit to page [[Wikipédia:Articles récents]] failed:
API write action attempted as IP u'Z\xe9roBot'
  File "maj_articles_recents.py", line 556, in main
    main_page.put(new_text, comment = comment)
  File "pywikibot/tools/__init__.py", line 1790, in wrapper
    return obj(*__args, **__kw)
  File "pywikibot/tools/__init__.py", line 1790, in wrapper
    return obj(*__args, **__kw)
  File "pywikibot/page.py", line 1379, in put
    **kwargs)
  File "pywikibot/tools/__init__.py", line 1790, in wrapper
    return obj(*__args, **__kw)
  File "pywikibot/tools/__init__.py", line 1790, in wrapper
    return obj(*__args, **__kw)
  File "pywikibot/page.py", line 1301, in save
    cc=apply_cosmetic_changes, quiet=quiet, **kwargs)
  File "pywikibot/page.py", line 134, in wrapper
    handle(func, self, *args, **kwargs)
  File "pywikibot/page.py", line 126, in handle
    raise pywikibot.OtherPageSaveError(self, err)

I don't know if these errors are excepted on the master branch.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptFeb 14 2020, 11:20 PM
JJMC89 added a subscriber: JJMC89.Feb 14 2020, 11:57 PM

For APIError: badinteger: Invalid value "3|0" for integer parameter "maxlag"., it looks like rPWBCa131de634c9d: [IMPR] use setdefault in api._add_defaults and https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2020-February/000152.html could be related.

Framawiki updated the task description. (Show Details)Feb 14 2020, 11:58 PM
Dvorapa added a comment.EditedFeb 14 2020, 11:58 PM

Hmmm, API write action attempted as IP u'Z\xe9roBot' seems weird. Another newly occurring login/logout issue lately.

@Framawiki Could you find also last time both these were working correctly without issues? Was it the previous day or a month ago (as last stable version is more than month old)?

I don't know if these errors are excepted on the master branch.

We should yet release another at least one Python 2 compatible and python2 tagged version, so any issue Python2/3 related, should still be a priority :)

I mean, the issues like this obviously help to force users to move a) to Python 3 b) to stable releases, but still I don't like the idea of being forced to Python 3 ( a) ).

Dvorapa added a comment.EditedFeb 14 2020, 11:59 PM

For APIError: badinteger: Invalid value "3|0" for integer parameter "maxlag"., it looks like rPWBCa131de634c9d: [IMPR] use setdefault in api._add_defaults and https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2020-February/000152.html could be related.

I see a mistake!

Change 572369 had a related patch set uploaded (by Dvorapa; owner: Dvorapa):
[pywikibot/core@master] [bugfix] Fix typo in a131de634c

https://gerrit.wikimedia.org/r/572369

Change 572369 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] Fix typo in a131de634c

https://gerrit.wikimedia.org/r/572369

Dvorapa removed a project: Patch-For-Review.EditedFeb 15 2020, 12:27 AM

Hmm, thank you for pointing this out, I haven't read this newsmail yet

Xqt renamed this task from Errors on Python2 starting February 12 to APIErrors for maxlag parameter higher than 9 starting February 12.Feb 15 2020, 12:57 AM
Xqt added a comment.Feb 15 2020, 12:59 AM

APIError: badinteger: Invalid value "3|0" for integer parameter "maxlag".

and we are broken by T242081 :-D

Xqt closed this task as Resolved.Feb 15 2020, 12:59 AM
Xqt claimed this task.
Xqt reopened this task as Open.Feb 15 2020, 1:04 AM

Remaining issue: API write action attempted as IP u'Z\xe9roBot'

Xqt removed Xqt as the assignee of this task.Feb 15 2020, 6:28 AM
Xqt added a comment.Feb 15 2020, 10:36 AM

@Framawiki: do you get any deprecation warning like 'ipaddr module or tools.ip.ip_regexp is deprecated; use ipaddress module instead' if you have showing DeprecationWarnings enabled? Do you have ipaddress packages installed for your Python 2 environment? Looks like there is a wrong result for user ZéroBot.

Change 572406 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [IMPR] use mw API IP/anon user detection

https://gerrit.wikimedia.org/r/572406

Xqt claimed this task.Feb 15 2020, 10:45 AM
Xqt triaged this task as Medium priority.

Change 572406 merged by jenkins-bot:
[pywikibot/core@master] [IMPR] use mw API IP/anon user detection

https://gerrit.wikimedia.org/r/572406

(ipaddress is surely installed in Toolforge at least)

Framawiki added a comment.EditedFeb 15 2020, 1:36 PM

Scripts run daily (and even multiple time a day). All scripts for this account were stuck during the problem period, even if they are started independently.
Not found ipaddress package so far in my venv.

tools.totoazero@tools-sgebastion-08:~$ ./venv/bin/python -V
Python 2.7.13

tools.totoazero@tools-sgebastion-08:~$ ./venv/bin/pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package           Version  
----------------- ---------
certifi           2017.11.5
chardet           3.0.4    
enum34            1.1.6    
idna              2.6      
mwoauth           0.3.2    
mysql             0.0.1    
MySQL-python      1.2.5    
oauthlib          2.0.6    
pip               19.3.1   
pkg-resources     0.0.0    
PyJWT             1.5.3    
requests          2.18.4   
requests-oauthlib 0.8.0    
sentry-sdk        0.12.2   
setuptools        40.8.0   
six               1.11.0   
urllib3           1.22     
wheel             0.33.1   
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

tools.totoazero@tools-sgebastion-08:~$ ./venv/bin/python
Python 2.7.13 (default, Sep 26 2018, 18:42:22) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ipaddress
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named ipaddress
>>>

tools.totoazero@tools-sgebastion-08:~$ crontab -l
50 23 * * * jsub -quiet -j y -o logs/new_log.out -N new_log mkdir -p logs/$(date --date='tomorrow' "+\%Y/\%m/\%d") 
0 */6 * * * jsub -mem 300m -quiet -j y -once -N maj_articles_recents -o logs/$(date "+\%Y/\%m/\%d")/maj_articles_recents.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/maj_articles_recents.py
#0 * * * * jsub -quiet -j y -once -N forum_nouveaux -o logs/$(date "+\%Y/\%m/\%d")/forum_nouveaux.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/forum_nouveaux.py
0 */2 * * * jsub -quiet -j y -once -N drp_warn -o logs/$(date "+\%Y/\%m/\%d")/drp_warn.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/drp_warn.py
##moved to framabot##0 */2 * * * jsub -quiet -j y -once -N unblock -o logs/$(date "+\%Y/\%m/\%d")/unblock.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/unblock.py
19 3 * * 1 jsub -quiet -j y -once -N modeles_sans_dates -o logs/$(date "+\%Y/\%m/\%d")/modeles_sans_dates.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/modeles_sans_dates.py
50 21 * * * jsub -quiet -j y -once -N stats_bistro -o logs/$(date "+\%Y/\%m/\%d")/stats_bistro.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/stats_bistro.py
03 2 * * * jsub -quiet -j y -once -N recreation -o logs/$(date "+\%Y/\%m/\%d")/recreation.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/recreation.py --prefix 'Utilisateur:ZéroBot/Journal_des_recréations'
10 2 * * * jsub -quiet -j y -once -N afd -o logs/$(date "+\%Y/\%m/\%d")/afd.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/afd.py -afdlist
23 2 * * * jsub -quiet -j y -once -N sysops_archives_new -o logs/$(date "+\%Y/\%m/\%d")/sysops_archives_new.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/sysops_archives_new.py
32 2 * * * jsub -quiet -j y -once -N sysops_archives -o logs/$(date "+\%Y/\%m/\%d")/sysops_archives.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/sysops_archives.py
32 2 * * * jsub -quiet -j y -once -N sysops_archives_renew -o logs/$(date "+\%Y/\%m/\%d")/sysops_archives_renew.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/sysops_archives_renew.py
#47 2 * * * jsub -quiet -j y -once -N spa -o logs/$(date "+\%Y/\%m/\%d")/spa.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/spa.py -full
26 1 * * 0 jsub -quiet -j y -once -N random_pages_portails -o logs/$(date "+\%Y/\%m/\%d")/random_pages_portails.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/random_pages_portails.py
30 1 */15 * * jsub -quiet -j y -once -N bistro -o logs/$(date "+\%Y/\%m/\%d")/bistro.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/bistro.py
#39 1 * * * jsub -quiet -j y -once -N ipday -o logs/$(date "+\%Y/\%m/\%d")/ipday.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/ipday.py
#42 3 * * * jsub -quiet -j y -once -N mort_recente -o logs/$(date "+\%Y/\%m/\%d")/mort_recente.py ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/mort_recente.py
42 3 * * * jsub -quiet -j y -N hotarticles -o logs/$(date "+\%Y/\%m/\%d")/hotarticles.out ./venv/bin/python /data/project/pywikibot/public_html/core_python2/pwb.py scripts-github/pywikibot/hotarticles.py
5 3 * * * jsub -quiet -j y -once -N mysqldump -o logs/$(date "+\%Y/\%m/\%d")/mysqldump.out bash sql-dump.sh
tools.totoazero@tools-sgebastion-08:~$

I also see depreciation warnings :

WARNING: /mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/data/api.py:1256: FutureWarning: ipaddr module or tools.ip.ip_regexp is deprecated for 24 days; use ipaddress module instead.
  if is_IP(self.site._userinfo['name']):

WARNING: /mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/tools/ip.py:77: FutureWarning: page.ip_regexp is deprecated for 5 years; use tools.is_IP instead.
  if ip_regexp.match(IP) is None:
Framawiki closed this task as Resolved.Feb 15 2020, 1:57 PM

Based on tests I've made, I can no longer reproduce username issue by applying @Xqt patch 7b21bc0, yeah! Thanks all of you, looks like we can close this task.

Thank you for your report and feel free to open a new one if any other issue occurs

Xqt added a comment.Feb 15 2020, 4:40 PM

I also see depreciation warnings :

I propose to install ipaddress package which is mandatory for Python 2.

Xqt added a comment.Feb 15 2020, 4:45 PM

Thank you for your report and feel free to open a new one if any other issue occurs

Yes, thank you @Framawiki for testing of mater branch. It was my fault. I haven't Python 2 installed anymore and cannot test it before commit.