Page MenuHomePhabricator

Drop support of Python 2.7
Closed, ResolvedPublic

Description

Python 2.7 as reached his EOL release and will not be maintained after January 1st, 2020 [1] due to PEP 373 [2]. More and more projects withdraw support of python 2.7 [3] including test packages [4]

I propose to use the full potental for python 3 and to cleanup code from old stuff. This hasn't to be immediately but bot owners should be conducted to Switch so python 3 soon.

What can be the steps to proceed?

  • Abandon support for Python 2.6 (T154771)
  • Drop support for Python 3.3 (T184508)
  • Drop support for python 2.7.2 and 2.7.3 now (T191192)
  • Drop support for Python 2.7.6 and lower in few months (T203471)
  • Drop support for Python 2.7.8 and lower in few months due to InsecureWarning of urllib3 [5]
  • Withdraw Python 2.7.2/2.7.3 from appveyor test
  • Add python 3.7 to test matrix gerrit:433554
  • Add python 3.8 to test matrix gerrit:482532
  • Drop support for python 2.7 in 2020
  • Provide a new "python2" tag to indicate a Python 2 stable release
  • Remove obsolete/deprecated/python2 related code
  • Release new major version

If someone cannot upgrade to Python 3.0, the older Pywikibot releases are still available either via pypi package or the corresponding tag in our repository but one should be aware that these issues may still lead to problems.

Newest test packages using Python 3 only

  • irc
  • pydocstyle (also used by other packages)
  • pytest (from version 5.0)
  • Pillow (from Version 7.0)

packages needed for Python 2.7 compatibility

  • enum34
  • future_buildins
  • future
  • ipaddress
  • mock
  • pathlib2
  • requests[security] (for 2.7.4 - 2.7.8)
  • stdnum
  • six
  • unicodecsv

[1] https://pythonclock.org/
[2] https://www.python.org/dev/peps/pep-0373/
[3] https://python3statement.org/
[4] https://pypi.org/project/pydocstyle/#description, T215874
[5] https://www.franzoni.eu/python-requests-ssl-and-insecureplatformwarning/

Related Objects

StatusSubtypeAssignedTask
ResolvedDalba
ResolvedDalba
OpenNone
OpenNone
OpenNone
ResolvedXqt
DeclinedNone
ResolvedXqt
DeclinedNone
ResolvedXqt
ResolvedXqt
ResolvedGoalXqt
ResolvedDvorapa
ResolvedLegoktm
OpenBUG REPORTNone
ResolvedBUG REPORTDvorapa
OpenNone
ResolvedBUG REPORTDvorapa
OpenNone
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedBUG REPORTDalba
ResolvedBUG REPORTXqt
ResolvedBUG REPORTXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedDalba
OpenBUG REPORTNone
ResolvedBUG REPORTXqt
ResolvedBUG REPORTNone
ResolvedBUG REPORTDzahn
ResolvedBUG REPORTDalba
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
OpenNone
ResolvedXqt
OpenNone
DeclinedNone
InvalidNone
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTXqt
ResolvedDvorapa
ResolvedDvorapa
OpenNone
OpenNone
ResolvedUrbanecm
OpenNone
ResolvedDvorapa
ResolvedBUG REPORTDvorapa
ResolvedDvorapa
OpenNone
OpenNone
ResolvedXqt
ResolvedXqt
OpenNone
ResolvedDvorapa
ResolvedXqt
ResolvedXqt
ResolvedXqt
OpenNone
ResolvedXqt
ResolvedXqt
ResolvedXqt
DeclinedNone
InvalidNone
ResolvedUrbanecm
ResolvedXqt
ResolvedXqt
DeclinedNone
Resolvedjayvdb
DeclinedNone
DuplicateNone
ResolvedDvorapa
ResolvedXqt
DeclinedNone
DeclinedNone
Resolvedhashar
ResolvedMpaa
ResolvedAndrew
DeclinedNone
ResolvedXqt
ResolvedDvorapa
ResolvedXqt
ResolvedDvorapa
DeclinedNone
ResolvedXqt
DeclinedNone
ResolvedDvorapa
ResolvedXqt
ResolvedXqt
DeclinedNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Xqt raised the priority of this task from Lowest to Low.Dec 1 2019, 6:10 PM
Legoktm added a subscriber: Legoktm.Jan 6 2020, 9:25 PM

EOL of Python 2.7 is also important for us because the most external libraries are also giving up Python 2 support (including pip btw). This means no further support, no bugfixes no new features. Ok you can say you never update libraries like requests or sseclient and what care about other stuff or about tests. On the other hand you don't need to update your bot. You can use every stable release submitted with their own tag; ask the HISTORY.rst file for their changes. Surely you will find a release running with Python 2.6. But this is not actually tested with current environments and external library dependencies.

I think this is a key factor in why we also will need to drop Python 2.7 support - the rest of the ecosystem we depend upon is also moving in that direction (https://python3statement.org/). At mwparserfromhell I'm looking at deleting ~500 lines of code to drop Python 2.7 support: https://github.com/earwig/mwparserfromhell/pull/234 - a huge win for future maintenance.

My proposal for how to move forward:

  • Set a date 3-4 months for now for the final pywikibot release that supports Python 2.7. I suggest we pick the same time as the final Python 2.7 release in April: https://www.python.org/psf/press-release/pr20191220/
    • After this date people can continue to use the old release or branch to run their bots, and any maintainers who'd like to step up to backport critical patches should be more than welcome to do so.
  • Set up a process (maybe a new phab project or a wiki page) in which script authors can post links to their code and receive assistance in Python 3 porting (usually mostly automatable with 2to3 and pyupgrade).

I don't think setting up a new branch for py3-only is a good idea, mostly because I would estimate that it requires *more* manual work on the part of everyone who already made their switch to Python 3 to help avoid breaking changes for the few people who are still on 2.7.

Set a date 3-4 months for now for the final pywikibot release that supports Python 2.7. I suggest we pick the same time as the final Python 2.7 release in April: https://www.python.org/psf/press-release/pr20191220/

Yeah, this is the option we discussed at WM Hackathon last year I think. But we planned to do it sooner actually.

Agree with @Legoktm, looks a good move.

But we planned to do it sooner actually.

I'm happy to do it earlier if others are on board :)

Xqt added a comment.Jan 7 2020, 6:34 AM

Strong agree. I propose to have the next stable release (i.e. after the tests passes) beeing the last one that support Python 2. Probably that will be in few months. The announcement dropping that release was long enough. I also like the idea of the 2to3 support.

Other important steps:

  • use semver versioning because our current outinatic increasing version number leads to problems with installing pwb.
  • cleanup code and remove old deprecated parts. This might be a breaking change but bot owners can be informed with FutureWarning instead of DeprecationWarning

But let us do this step dropping py2 first and let Travis and Appveyor pass again

See also my comment https://gerrit.wikimedia.org/r/#/c/pywikibot/core/+/562032/

Xqt raised the priority of this task from Low to Medium.Jan 7 2020, 6:35 AM
Xqt updated the task description. (Show Details)Jan 7 2020, 6:38 AM
Xqt updated the task description. (Show Details)

Okay, let's focus on T225591 now.

About that versioning: After that last release are we going to move to PEP 440? And after that we can increment versions like 4.0.0-pre1, -pre2, until we clean code and after that, push 4.0.0?

Xqt added a comment.Jan 7 2020, 3:47 PM

I've introduced a new tag python2 which marks the last stable release für Python 2.7,
see https://gerrit.wikimedia.org/r/#/admin/projects/pywikibot/core,tags

Per T242157 we should inform users as much as possible.

We could inspire ourselves with the timelines for Trusty deprecation: https://wikitech.wikimedia.org/wiki/News/Trusty_deprecation#Cloud_VPS_projects and https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation#Timeline

First we should announce the 2.7 deprecation in mailing lists ASAP

Dvorapa updated the task description. (Show Details)Jan 13 2020, 1:51 PM
Xqt updated the task description. (Show Details)Jan 13 2020, 5:32 PM
Dvorapa moved this task from Backlog to Announced on the Pywikibot-Announce board.Jan 20 2020, 1:35 AM

First phase announced in mailing lists and tech news

Xqt added a comment.EditedJan 20 2020, 5:34 AM

First phase announced in mailing lists and tech news

Did you? Didn’t received sth
Found it but why I didn’t received that mail from mailing list?

Xqt added a comment.EditedJan 20 2020, 5:54 AM

Seems we have to hurry up because ci tests for Python 2 cannot run at Appveyor anymore (T243133)

Dvorapa added a comment.EditedJan 20 2020, 8:57 AM

We can either ignore/hide the warning using PYTHONWARNINGS=ignore:::pkg_resources.py2_warn environment variable, or reinstall setuptools < 45

First phase announced in mailing lists and tech news

Did you? Didn’t received sth
Found it but why I didn’t received that mail from mailing list?

I have definitely sent it with multiple lists in To variable. It's in archives, so I blindly assume it got distributed. Little to none way to check for me.

Maybe deduplication at mailman level got wrong? Have you got at least one announcement email?

I have also included the info in tech news and WD weekly summary.

Xqt added a comment.Jan 20 2020, 2:37 PM

I have definitely sent it with multiple lists in To variable. It's in archives, so I blindly assume it got distributed. Little to none way to check for me.

Got it. Thanks.

Mentioned in SAL (#wikimedia-cloud) [2020-02-13T21:57:59Z] <Framawiki> edited crons to use new specific py2 pwb.py endpoint since T213287

Mentioned in SAL (#wikimedia-cloud) [2020-02-13T21:57:59Z] <Framawiki> edited crons to use new specific py2 pwb.py endpoint since T213287

Related to specific bot, please ignore

Xqt updated the task description. (Show Details)Feb 14 2020, 5:37 PM
Dvorapa renamed this task from Drop support of python 2.7 to Drop support of Python 2.7.Mar 4 2020, 6:44 PM

Change 594106 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [doc] The next tagged release is the last supporting Python 2 and 3.4

https://gerrit.wikimedia.org/r/594106

To those who don't watch subtasks: I've looked into how much users use Python2 those days, and uploaded the result as F31801803.

Change 594106 merged by jenkins-bot:
[pywikibot/core@master] [doc] The next tagged release is the last supporting Python 2

https://gerrit.wikimedia.org/r/594106

Change 609486 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [4.0] Prepare next release 4.0

https://gerrit.wikimedia.org/r/609486

Change 609497 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] Remove Python 3 backports from TestCaseBase

https://gerrit.wikimedia.org/r/609497

Change 609499 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] Remove Python 2 parts from utils.py

https://gerrit.wikimedia.org/r/609499

Change 609486 merged by jenkins-bot:
[pywikibot/core@master] [4.0] Remove Python 2 dependencies

https://gerrit.wikimedia.org/r/609486

Change 610062 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [4.0] Remove code duplication of pwb.py and setup.py

https://gerrit.wikimedia.org/r/610062

Change 610295 had a related patch set uploaded (by Xqt; owner: Xqt):
[integration/config@master] [tests] use Python 3.7 for Jenkins tests

https://gerrit.wikimedia.org/r/610295

Change 609497 merged by jenkins-bot:
[pywikibot/core@master] [4.0] Remove Python 3 backports from TestCaseBase

https://gerrit.wikimedia.org/r/609497

Change 610062 merged by jenkins-bot:
[pywikibot/core@master] [4.0] Remove code duplication of pwb.py and setup.py

https://gerrit.wikimedia.org/r/610062

Change 610791 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [4.0] Remove Python 2 code from pywikibot/__init__.py

https://gerrit.wikimedia.org/r/610791

Xqt updated the task description. (Show Details)Jul 9 2020, 3:26 PM

Change 609499 merged by jenkins-bot:
[pywikibot/core@master] [tests] Remove Python 2 parts from utils.py

https://gerrit.wikimedia.org/r/609499

Change 610791 merged by jenkins-bot:
[pywikibot/core@master] [4.0] Remove Python 2 code from pywikibot/__init__.py

https://gerrit.wikimedia.org/r/610791

Change 611255 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Add pywikibot-core-tox-fasttest-py37-docker

https://gerrit.wikimedia.org/r/611255

Change 611256 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [pywikibot/core] Add fasttest-py37 as experimental

https://gerrit.wikimedia.org/r/611256

Change 611257 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [pywikibot/core] Add fasttest-py37 as voting

https://gerrit.wikimedia.org/r/611257

Change 611258 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Drop pywikibot-core-tox-fasttest-py27-docker, no longer used

https://gerrit.wikimedia.org/r/611258

Change 611255 merged by jenkins-bot:
[integration/config@master] jjb: Add pywikibot-core-tox-fasttest-py37-docker

https://gerrit.wikimedia.org/r/611255

Change 611256 merged by jenkins-bot:
[integration/config@master] layout: [pywikibot/core] Add fasttest-py37 as experimental

https://gerrit.wikimedia.org/r/611256

Change 612704 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[pywikibot/core@master] [4.0] Remove py2 mention from documentation

https://gerrit.wikimedia.org/r/612704

Change 612704 merged by jenkins-bot:
[pywikibot/core@master] [4.0] Remove py2 mention from documentation

https://gerrit.wikimedia.org/r/612704

Xqt closed this task as Resolved.Aug 4 2020, 11:28 AM
Xqt claimed this task.
Xqt updated the task description. (Show Details)

Change 611257 merged by jenkins-bot:
[integration/config@master] layout: [pywikibot/core] Add fasttest-py37 as voting

https://gerrit.wikimedia.org/r/611257

Change 610295 merged by jenkins-bot:
[integration/config@master] layout: [pywikibot/core] Drop fasttest-py27

https://gerrit.wikimedia.org/r/610295

Mentioned in SAL (#wikimedia-releng) [2020-08-12T17:00:07Z] <James_F> [pywikibot/core] Add fasttest-py37; drop fasttest-py27 T213287

Change 611258 merged by jenkins-bot:
[integration/config@master] jjb: Drop pywikibot-core-tox-fasttest-py27-docker, no longer used

https://gerrit.wikimedia.org/r/611258