Erfgoedbot runs has dozens of such warnings:
Description
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Bump pywikibot to latest version 3.0.20180823 | labs/tools/heritage | master | +21 -21 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dalba | T200955 Update pywikibot queries for MCR | |||
| Resolved | JeanFred | T202378 Erfgoedbot bot runs triggers lots of 'rvslots' API warnings |
Event Timeline
Per rPWBC057a50bf8e23 this is fixed in master.
Latest pywikibot release on Pypi is 3.0.20180710. We would need to wait for the next pywikibot release − not sure what the release cycle is…
This should hopefully also silence this warning:
For what it's worth I've appended the following script to my log-analysing shell script
https://gist.github.com/lokal-profil/ce3625346a84474b842715084b64175c
FYI an up-to-date version of PWB is available at /shared/pywikibot/core/pwb.py on tools hosts
We had problems running on this before since not pinning a version meant local, CI and production environments ended up differing. See T151783 and the task which led to it.
Pywikibot is released almost once a month, but it mostly depends on Xqt, the de facto release manager of pywikibot...
A new version is now on available on PYPI.
Change 455016 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Bump pywikibot to latest version 3.0.20180823
Change 455016 merged by jenkins-bot:
[labs/tools/heritage@master] Bump pywikibot to latest version 3.0.20180823
Mentioned in SAL (#wikimedia-cloud) [2018-09-01T21:48:04Z] <Lokal-Profil> Deploy latest from Git master: b7a9c43 (T202378)
++ pip install -r /data/project/heritage/heritage/requirements.txt
Exception:
Traceback (most recent call last):
File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/commands/install.py", line 262, in run
for req in parse_requirements(filename, finder=finder, options=options, session=session):
File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/req.py", line 1631, in parse_requirements
req = InstallRequirement.from_line(line, comes_from, prereleases=getattr(options, "pre", None))
File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/req.py", line 172, in from_line
return cls(req, comes_from, url=url, prereleases=prereleases)
File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/req.py", line 70, in __init__
req = pkg_resources.Requirement.parse(req)
File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2606, in parse
reqs = list(parse_requirements(s))
File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2544, in parse_requirements
line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
File "/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2522, in scan_list
"Expected ',' or end-of-list in",line,"at",line[p:]
ValueError: ("Expected ',' or end-of-list in", "urllib3==1.23; python_version >= '2.6'", 'at', "; python_version >= '2.6'")Soooo, the pip used on Toolforge is not recent enough for the requirements syntax spitted out by pipenv ? Yaaaaay...
Oh noes...
(.venv)tools.heritage@tools-bastion-03:~$ tail -n 100 -f logs/categorize_images.log
2018-09-01_21:51:49 Categorize images...
WARNING: /data/project/heritage/pywikibot/pywikibot/data/api.py:310: UserWarning: Unexpected overlap between action and query submodules: frozenset([u'globalpreferences', u'readinglists'])
WARNING: /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:369: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning
WARNING: /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:160: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarning
Retrieving 11 pages from commons:commons.
WARNING: API warning (main): Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application.
WARNING: API warning (revisions): Because "rvslots" was not specified, a legacy format has been used for the output. This format is deprecated, and in the future the new format will always be used.
Traceback (most recent call last):
File "/data/project/heritage/heritage/erfgoedbot/categorize_images.py", line 681, in <module>
main()
File "/data/project/heritage/heritage/erfgoedbot/categorize_images.py", line 671, in main
cursor)
File "/data/project/heritage/heritage/erfgoedbot/categorize_images.py", line 489, in processCountry
'cat': commonsCategoryBase.title(with_ns=False),
File "/data/project/heritage/pywikibot/pywikibot/tools/__init__.py", line 1448, in wrapper
return obj(*__args, **__kw)
TypeError: title() got an unexpected keyword argument 'with_ns'S**t. So emergency patch changing that first line of the requirements.txt file followed by an upstream task for bumping pip?
It’s actually the last one, the > python 2.6. Just changed it manually on the server >_>
But now we have bigger problems...
I'm guessing that second one is because pywikibot didn't update but the change to the new parameter names did?
…
Installing Python requirements...
+ pip install -r /data/project/heritage/heritage/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): certifi==2018.8.24 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): chardet==3.0.4 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): future==0.16.0 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): idna==2.7 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): ipaddr==2.2.0 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): pymysql==0.7.11 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 7))
Downloading/unpacking pywikibot==3.0.20180823 (from -r /data/project/heritage/heritage/requirements.txt (line 8))
Downloading pywikibot-3.0.20180823.tar.gz (511kB): 511kB downloaded
Running setup.py (path:/mnt/nfs/labstore-secondary-tools-project/heritage/.venv/build/pywikibot/setup.py) egg_info for package pywikibot
fatal: Not a git repository (or any parent up to mount point /mnt/nfs/labstore-secondary-tools-project)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
Command '[u'git', u'log', u'-1', u'--format=%ci']' returned non-zero exit status 128
Cannot include pywikibot/DIRECTORIES.rst; file not found
Cannot include HISTORY.rst; file not found
Cannot include CODE_OF_CONDUCT.rst; file not found
Requested pywikibot==3.0.20180823 (from -r /data/project/heritage/heritage/requirements.txt (line 8)), but installing version 3.0.dev0
Requirement already satisfied (use --upgrade to upgrade): MySQL-python in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): pyyaml==3.13 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 10))
Requirement already satisfied (use --upgrade to upgrade): requests==2.19.1 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): urllib3==1.23 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from -r /data/project/heritage/heritage/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): pycparser!=2.14 in /mnt/nfs/labstore-secondary-tools-project/heritage/.venv/lib/python2.7/site-packages (from pywikibot==3.0.20180823->-r /data/project/heritage/heritage/requirements.txt (line 8))
Installing collected packages: pywikibot
Found existing installation: pywikibot 2.0rc2
Not uninstalling pywikibot at /mnt/nfs/labstore-secondary-tools-project/heritage/pywikibot, outside environment /mnt/nfs/labstore-secondary-tools-project/heritage/.venv
Running setup.py install for pywikibot
fatal: Not a git repository (or any parent up to mount point /mnt/nfs/labstore-secondary-tools-project)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
Command '[u'git', u'log', u'-1', u'--format=%ci']' returned non-zero exit status 128
Cannot include pywikibot/DIRECTORIES.rst; file not found
Cannot include HISTORY.rst; file not found
Cannot include CODE_OF_CONDUCT.rst; file not found
Successfully installed pywikibot
Cleaning up...From the log at T202378#4551191 it looks like its running /data/project/heritage/pywikibot/pywikibot/
Very good call @Lokal_Profil :) I updated pywikibot (git fetch && git checkout 3.0.20180823) and things are going through now. Pfew!
Same as on my machine. Whatever venv I run it always uses my local dev pywikibot. Which works great until it doesn't because I'm half way to reading some commit to pywikibot.
Is this actually the pip in the venv that is old? If so we should be able to upgrade that.
Latest pywikibot release on Pypi is 3.0.20180710. We would need to wait for the next pywikibot release − not sure what the release cycle is…
usually once a mont after all tests passed sucessfully. I deployed the new release today:
https://pypi.org/project/pywikibot/#history