Page MenuHomePhabricator

Install numerious python packages from stretch-backports for compatibility with pywikibot
Closed, ResolvedPublic

Description

  • python-requests
  • python-ipaddress
  • python-pathlib2
  • python3-requests

Steps to reproduce

  1. run python3 /shared/pywikibot/core/pwb.py add_text

Expected behavior
It should correctly run the script.

Current behavior
Toolforge bots using Pywikibot master (instead of stable, silly) do not work currently.

(requests 2.12.4 (/usr/lib/python3/dist-packages), Requirement.parse('requests>=2.20.1; python_version != "3.4"'))

Please install/update required module with:


    pip install "requests>=2.20.1"

Further details
It seems Toolforge chose default Debian requests over Wikimedia one:

$ apt-cache policy python3-requests
python3-requests:
  Installed: 2.12.4-1
  Candidate: 2.12.4-1
  Version table:
     2.21.0-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
 *** 2.12.4-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

The same for Python 2 and python-requests and python-ipaddress.
Not sure how to fix this.

Event Timeline

Dvorapa renamed this task from Toolforge switched to old version of requests to Toolforge switched to old version of python(3)-requests.Mar 24 2020, 8:24 AM
Dvorapa updated the task description. (Show Details)
Dvorapa updated the task description. (Show Details)
Dvorapa updated the task description. (Show Details)

If this is not the cause, would it be able to upgrade python(3)-requests to v2.20.1+? Pywikibot requires this version and thus can't work on Toolforge currently.

Dvorapa renamed this task from Toolforge switched to old version of python(3)-requests to Toolforge chooses old version of python(3)-requests.Mar 24 2020, 8:32 AM

requests 2.12.4 is 40 months old and very outdated. There are some external dependencies like urllib3, idna, chardet, certifi and I've no glue whether this is related to Python versions. Anyway requests 2.12.4 is not compatible with Python 3.7. Refer https://requests.readthedocs.io/en/master/community/updates/#id4 for other importand changes of requests.

The lowest request version needed for sseclient is 2.9 and 2.18.2 causes bug T172059.

The same issue with ipaddress on Python 2:

$ python pwb.py login
(requests 2.12.4 (/usr/lib/python2.7/dist-packages), Requirement.parse('requests>=2.20.1; python_version != "3.4"'))
(ipaddress 1.0.17 (/usr/lib/python2.7/dist-packages), Requirement.parse('ipaddress>=1.0.23; python_version < "3"'))

Please install/update required modules with:


    pip install "requests>=2.20.1"

    pip install "ipaddress>=1.0.23"

$ apt-cache policy python-ipaddress
python-ipaddress:
  Installed: 1.0.17-1
  Candidate: 1.0.17-1
  Version table:
 *** 1.0.17-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

@Xqt Perhaps we should allow ipaddress 1.0.17 because of Toolforge?

(Could we also change this error to just a warning? Because Toolforge has some other Pywikibot dependencies too old)

Dvorapa updated the task description. (Show Details)
Dvorapa updated the task description. (Show Details)
Dvorapa updated the task description. (Show Details)

Pywikibot master (instead of stable, silly)

I run selected tasks on master branch to detect regressions and other bugs, not a silly thing I guess :P

It looks like in this particular case we can force installing the package from stretch-backports as a near term fix.

The permanent fix for this is to use a virtualenv for your Python library dependencies rather than relying on the system packages. The system packages are installed via Debian packages and thus they will always be older versions.

bd808 renamed this task from Toolforge chooses old version of python(3)-requests to Install python3-requests from stretch-backports for compatibility with pywikibot master.Mar 24 2020, 2:24 PM
bd808 edited projects, added Toolforge (Software install/update); removed Toolforge.

Pywikibot master (instead of stable, silly)

I run selected tasks on master branch to detect regressions and other bugs, not a silly thing I guess :P

For detection of bugs and regressions not, I admit :D (I do it too). Not good for production where you really need it to work and be maintainless.

It looks like in this particular case we can force installing the package from stretch-backports as a near term fix.

The permanent fix for this is to use a virtualenv for your Python library dependencies rather than relying on the system packages. The system packages are installed via Debian packages and thus they will always be older versions.

python3-requests is the only and most important Pywikibot dependency for Python 3, I would vote for stretch-backports in this case.

python-requests and python-ipaddress are Pywikibot dependencies for Python 2. Its support will be ended in Pywikibot soon, so their update is not crucial. Some Python 2 Pywikibot bots left in Toolforge broke by this strict dependency check, some will break with the next Pywikibot release. Only venv bots should be fine. The question is if we want to break any. I would say yes :D but I know so many people will argue with me.

For everyone with broken bot: You can (should) use Pywikibot stable (/shared/pywikibot/stable or git tag stable), where this issue does not occur. See comments above

Change 583293 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] Enable pywikibot run even there is a package version conflict

https://gerrit.wikimedia.org/r/583293

(Could we also change this error to just a warning? Because Toolforge has some other Pywikibot dependencies too old)

Did it. The script will be run even there is a version conflict but gives a warning. Should we have a config variable like strict_dependencies to let the script fail if there is a version conflict?

Pywikibot master (instead of stable, silly)

I run selected tasks on master branch to detect regressions and other bugs, not a silly thing I guess :P

Yes, we need experienced users which helps reviewing. Thanks.
PAWS uses stable btw.

Change 583293 merged by jenkins-bot:
[pywikibot/core@master] Enable pywikibot run even there is a package version conflict

https://gerrit.wikimedia.org/r/583293

For everyone with broken bot: You can (should) use Pywikibot stable (/shared/pywikibot/stable or git tag stable), where this issue does not occur. See comments above

It works! Thank you.

Dvorapa renamed this task from Install python3-requests from stretch-backports for compatibility with pywikibot master to Install python3-requests from stretch-backports for compatibility with pywikibot.Mar 25 2020, 4:36 PM

OK. I set up all our processes in our crontab to /shared/pywikibot/stable. It was fine everything until today. Now I see on the log of atobot again as follows:

(requests 2.12.4 (/usr/lib/python3/dist-packages), Requirement.parse('requests>=2.20.1; python_version != "3.4"'))

A package necessary for pywikibot is outdated.
Please update required module with:

pip install "requests>=2.20.1"

You may continue on your own risk; type CTRL-C to stop.

Yes, but it should be only a warning now, not a failure. Pywikibot should continue after sleep(5s)

On the Toolforge side, we install the 'python-requests' and 'python3-requests' packages as part of the profile::toolforge::genpp::python_exec_stretch Puppet module. That module is generated code, so we will need to think a bit about how to properly add pinning there so that the pinning survives subsequent runs of the module generation script.

Also for archivebot.py jobs the script fails completely. I am getting this error on the grid

Traceback (most recent call last):
  File "scripts/archivebot.py", line 111, in <module>
    import pywikibot
  File "/shared/pywikipedia/core/pywikibot/__init__.py", line 25, in <module>
    from pywikibot.bot import (
  File "/shared/pywikipedia/core/pywikibot/bot.py", line 109, in <module>
    from pathlib2 import Path
ImportError: No module named pathlib2

If it helps it is in the cronjob for Wikisource-bot

bd808 renamed this task from Install python3-requests from stretch-backports for compatibility with pywikibot to Install numerious python packages from stretch-backports for compatibility with pywikibot.Apr 9 2020, 2:33 AM
bd808 updated the task description. (Show Details)
bd808 triaged this task as High priority.Apr 9 2020, 2:37 AM

I'm honestly not super happy about pushing this to high priority and installing new global Python packages, but I do recognize that a large amount of the work done for the wikis on the Toolforge grid engine is dependent on pywikibot.

Please note that for running scripts with Python 2 the following packages are mandatory:

  • enum34>=1.1.6,!=1.1.8
  • ipaddress>=1.0.23
  • pathlib2
  • requests>=2.20.1

Supporting Python 2 is deprecated and will be dropped soon (T213287).

For Python 3 we only have

  • requests>=2.20.1

Supporting Python 3.4 is deprecated and will be dropped soon (T239542).

Several scripts might have additional dependencies.

Please note that for running scripts with Python 2 the following packages are mandatory:

  • enum34>=1.1.6,!=1.1.8
$ apt-cache policy python-enum34
python-enum34:
  Installed: 1.1.6-1
  Candidate: 1.1.6-1
  Version table:
 *** 1.1.6-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
  • ipaddress>=1.0.23

1.0.17 is the best we can do with upstream packaging on the Toolforge bastions and grid engine nodes (Debian Stretch). The actual code delta from 1.0.17 to 1.0.23 looks pretty minor -- https://github.com/phihag/ipaddress/compare/v1.0.17...master

$ apt-cache policy python-ipaddress
python-ipaddress:
  Installed: 1.0.17-1
  Candidate: 1.0.17-1
  Version table:
 *** 1.0.17-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
  • pathlib2

This one must currently be a dependency of one of the packages that we only install on the bastions. Adding to the grid nodes should be simple.

$ apt-cache policy python-pathlib2
python-pathlib2:
  Installed: 2.2.0+really2.1.0-1
  Candidate: 2.2.0+really2.1.0-1
  Version table:
 *** 2.2.0+really2.1.0-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
  • requests>=2.20.1

We need to pin this one to stretch-backports for a working version.

$ apt-cache policy python-requests
python-requests:
  Installed: 2.12.4-1
  Candidate: 2.12.4-1
  Version table:
     2.21.0-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
 *** 2.12.4-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

Supporting Python 2 is deprecated and will be dropped soon (T213287).

T248914: Information campaign and tech support needed to get Toolforge pywikibot use moved from py2 to py3 -- I don't think the Toolforge community has been well informed of this yet.

For Python 3 we only have

  • requests>=2.20.1

We need to pin this one to stretch-backports for a working version.

$ apt-cache policy python3-requests
python3-requests:
  Installed: 2.12.4-1
  Candidate: 2.12.4-1
  Version table:
     2.21.0-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
 *** 2.12.4-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

Supporting Python 3.4 is deprecated and will be dropped soon (T239542).

The only place we have py3.4 in Toolforge is on the Kubernetes cluster via an old Jessie based image. We are hoping to deprecate that before July 2020. At the time I write this there are 107 pods on the Kubernetes cluster using the python3.4 image. It is actually scarily the 2nd most popular image. Php5 is the most popular and also one we need to deprecate.

Change 587894 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: Update python dependencies for pywikibot

https://gerrit.wikimedia.org/r/587894

bd808 moved this task from Backlog to Job grid on the Toolforge (Software install/update) board.
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 587894 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: Update python dependencies for pywikibot

https://gerrit.wikimedia.org/r/587894

Change 587979 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [deps] Do not restrict ipaddress version dependency

https://gerrit.wikimedia.org/r/587979

Change 587979 merged by jenkins-bot:
[pywikibot/core@master] [deps] Do not restrict ipaddress version dependency

https://gerrit.wikimedia.org/r/587979

$ ssh dev.toolforge.org
$ become bd808-pywikibot
$ python3 /shared/pywikibot/core/pwb.py add_text
ERROR: Unable to execute script because no generator was defined.
The text to add wasn't given.
Use -help for further information.
$ python /shared/pywikibot/core/pwb.py add_text
WARNING: /mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/__init__.py:128: FutureWarning:

Python 2.7.13 will be dropped soon.
It is recommended to use Python 3.5 or above.
See T213287 for further information.

  FutureWarning)  # probably adjust the line no in utils.execute()

ERROR: Unable to execute script because no generator was defined.
The text to add wasn't given.
Use -help for further information.