Page MenuHomePhabricator

Erfgoedbot bot runs triggers lots of 'rvslots' API warnings
Closed, ResolvedPublic

Description

Erfgoedbot runs has dozens of such warnings:

WARNING: API warning (main): Subscribe to the mediawiki-api-announce mailing list at https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce for notice of API deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application.
WARNING: API warning (revisions): Because "rvslots" was not specified, a legacy format has been used for the output. This format is deprecated, and in the future the new format will always be used.

Details

Event Timeline

Per rPWBC057a50bf8e23 this is fixed in master.

Latest pywikibot release on Pypi is 3.0.20180710. We would need to wait for the next pywikibot release − not sure what the release cycle is…

This should hopefully also silence this warning:

WARNING: /data/project/heritage/pywikibot/pywikibot/data/api.py:310: UserWarning: Unexpected overlap between action and query submodules: frozenset([u'globalpreferences', u'readinglists'])

For what it's worth I've appended the following script to my log-analysing shell script
https://gist.github.com/lokal-profil/ce3625346a84474b842715084b64175c

Per rPWBC057a50bf8e23 this is fixed in master.

Latest pywikibot release on Pypi is 3.0.20180710. We would need to wait for the next pywikibot release − not sure what the release cycle is…

FYI an up-to-date version of PWB is available at /shared/pywikibot/core/pwb.py on tools hosts

Per rPWBC057a50bf8e23 this is fixed in master.

Latest pywikibot release on Pypi is 3.0.20180710. We would need to wait for the next pywikibot release − not sure what the release cycle is…

FYI an up-to-date version of PWB is available at /shared/pywikibot/core/pwb.py on tools hosts

We had problems running on this before since not pinning a version meant local, CI and production environments ended up differing. See T151783 and the task which led to it.

Latest pywikibot release on Pypi is 3.0.20180710. We would need to wait for the next pywikibot release − not sure what the release cycle is…

Pywikibot is released almost once a month, but it mostly depends on Xqt, the de facto release manager of pywikibot...
A new version is now on available on PYPI.

Change 455016 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Bump pywikibot to latest version 3.0.20180823

https://gerrit.wikimedia.org/r/455016

Change 455016 merged by jenkins-bot:
[labs/tools/heritage@master] Bump pywikibot to latest version 3.0.20180823

https://gerrit.wikimedia.org/r/455016

Mentioned in SAL (#wikimedia-cloud) [2018-09-01T21:48:04Z] <Lokal-Profil> Deploy latest from Git master: b7a9c43 (T202378)

++ pip install -r /data/project/heritage/heritage/requirements.txt

Exception:
Traceback (most recent call last):
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/commands/install.py", line 262, in run
    for req in parse_requirements(filename, finder=finder, options=options, session=session):
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/req.py", line 1631, in parse_requirements
    req = InstallRequirement.from_line(line, comes_from, prereleases=getattr(options, "pre", None))
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/req.py", line 172, in from_line
    return cls(req, comes_from, url=url, prereleases=prereleases)
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/req.py", line 70, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2606, in parse
    reqs = list(parse_requirements(s))
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2544, in parse_requirements
    line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
  File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2522, in scan_list
    "Expected ',' or end-of-list in",line,"at",line[p:]
ValueError: ("Expected ',' or end-of-list in", "urllib3==1.23; python_version >= '2.6'", 'at', "; python_version >= '2.6'")

Soooo, the pip used on Toolforge is not recent enough for the requirements syntax spitted out by pipenv ? Yaaaaay...

Oh noes...

(.venv)tools.heritage@tools-bastion-03:~$ tail -n 100 -f logs/categorize_images.log
2018-09-01_21:51:49 Categorize images...

WARNING: /data/project/heritage/pywikibot/pywikibot/data/api.py:310: UserWarning: Unexpected overlap between action and query submodules: frozenset([u'globalpreferences', u'readinglists'])
WARNING: /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:369: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning

WARNING: /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:160: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning

Retrieving 11 pages from commons:commons.
WARNING: API warning (main): Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application.
WARNING: API warning (revisions): Because "rvslots" was not specified, a legacy format has been used for the output. This format is deprecated, and in the future the new format will always be used.
Traceback (most recent call last):
  File "/data/project/heritage/heritage/erfgoedbot/categorize_images.py", line 681, in <module>
    main()
  File "/data/project/heritage/heritage/erfgoedbot/categorize_images.py", line 671, in main
    cursor)
  File "/data/project/heritage/heritage/erfgoedbot/categorize_images.py", line 489, in processCountry
    'cat': commonsCategoryBase.title(with_ns=False),
  File "/data/project/heritage/pywikibot/pywikibot/tools/__init__.py", line 1448, in wrapper
    return obj(*__args, **__kw)
TypeError: title() got an unexpected keyword argument 'with_ns'

S**t. So emergency patch changing that first line of the requirements.txt file followed by an upstream task for bumping pip?

S**t. So emergency patch changing that first line of the requirements.txt file followed by an upstream task for bumping pip?

It’s actually the last one, the > python 2.6. Just changed it manually on the server >_>

But now we have bigger problems...

I'm guessing that second one is because pywikibot didn't update but the change to the new parameter names did?

Installing Python requirements...
+ pip install -r /data/project/heritage/heritage/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): certifi==2018.8.24 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): chardet==3.0.4 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): future==0.16.0 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): idna==2.7 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): ipaddr==2.2.0 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): pymysql==0.7.11 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 7))
Downloading/unpacking pywikibot==3.0.20180823 (from -r /data/project/heritage/heritage/requirements.txt (line 8))
  Downloading pywikibot-3.0.20180823.tar.gz (511kB): 511kB downloaded
  Running setup.py (path:/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/build/pywikibot/setup.py) egg_info for package pywikibot
    fatal: Not a git repository (or any parent up to mount point /mnt/nfs/labstore-secondary-tools-project)
    Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)
    Command '[u'git', u'log', u'-1', u'--format=%ci']' returned non-zero exit status 128
    Cannot include pywikibot/DIRECTORIES.rst; file not found
    Cannot include HISTORY.rst; file not found
    Cannot include CODE_OF_CONDUCT.rst; file not found
    
  Requested pywikibot==3.0.20180823 (from -r /data/project/heritage/heritage/requirements.txt (line 8)), but installing version 3.0.dev0
Requirement already satisfied (use --upgrade to upgrade): MySQL-python in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): pyyaml==3.13 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 10))
Requirement already satisfied (use --upgrade to upgrade): requests==2.19.1 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): urllib3==1.23 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): pycparser!=2.14 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from pywikibot==3.0.20180823->-r /data/project/heritage/heritage/requirements.txt (line 8))
Installing collected packages: pywikibot
  Found existing installation: pywikibot 2.0rc2
    Not uninstalling pywikibot at /mnt/nfs/labstore-secondary-tools-project/heritage/pywikibot, outside environment /mnt/nfs/labstore-secondary-tools-project/heritage/.venv
  Running setup.py install for pywikibot
    fatal: Not a git repository (or any parent up to mount point /mnt/nfs/labstore-secondary-tools-project)
    Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)
    Command '[u'git', u'log', u'-1', u'--format=%ci']' returned non-zero exit status 128
    Cannot include pywikibot/DIRECTORIES.rst; file not found
    Cannot include HISTORY.rst; file not found
    Cannot include CODE_OF_CONDUCT.rst; file not found
    
Successfully installed pywikibot
Cleaning up...

Have we been doing a local git update to the /pywikbot directory?

Have we been doing a local git update to the /pywikbot directory?

Not sure I follow − everything should be run from the virtual environment no ?

From the log at T202378#4551191 it looks like its running /data/project/heritage/pywikibot/pywikibot/

Very good call @Lokal_Profil :) I updated pywikibot (git fetch && git checkout 3.0.20180823) and things are going through now. Pfew!

I totally don’t understand how come that pywkibot is used, but oh well :)

I totally don’t understand how come that pywkibot is used, but oh well :)

Same as on my machine. Whatever venv I run it always uses my local dev pywikibot. Which works great until it doesn't because I'm half way to reading some commit to pywikibot.

Soooo, the pip used on Toolforge is not recent enough for the requirements syntax spitted out by pipenv ? Yaaaaay...

Is this actually the pip in the venv that is old? If so we should be able to upgrade that.

Soooo, the pip used on Toolforge is not recent enough for the requirements syntax spitted out by pipenv ? Yaaaaay...

Is this actually the pip in the venv that is old? If so we should be able to upgrade that.

T203432: Toolforge pip cannot handle the requirements as formatted by pipenv

Xqt triaged this task as High priority.EditedSep 6 2018, 4:08 PM
Xqt subscribed.

Get it too recently with pywikibot master branch

Latest pywikibot release on Pypi is 3.0.20180710. We would need to wait for the next pywikibot release − not sure what the release cycle is…

usually once a mont after all tests passed sucessfully. I deployed the new release today:
https://pypi.org/project/pywikibot/#history