Page MenuHomePhabricator

Drop support of python 2.7
Open, LowestPublic

Description

Python 2.7 as reached his EOL release and will not be maintained after January 1st, 2020 [1] due to PEP 373 [2]. More and more projects withdraw support of python 2.7 [3] including test packages [4]

I propose to use the full potental for python 3 and to cleanup code from old stuff. This hasn't to be immediately but bot owners should be conducted to Switch so python 3 soon.

What can be the steps to proceed?

  • Abandon support for Python 2.6 (T154771)
  • Drop support for Python 3.3 (T184508)
  • Drop support for python 2.7.2 and 2.7.3 now (T191192)
  • Drop support for Python 2.7.6 and lower in few months (T203471)
  • Drop support for Python 2.7.8 and lower in few months due to InsecureWarning of urllib3 [5]
  • Withdraw Python 2.7.2/2.7.3 from appveyor test
  • Add python 3.7 to test matrix gerrit:433554
  • Add python 3.8 to test matrix gerrit:482532
  • New functionalities for python 3 only
  • Drop support for python 2.7 in 2020

If someone cannot upgrade to Python 3.0, the older Pywikibot releases are still available either via pypi package or the corresponding tag in our repository but one should be aware that these issues may still lead to problems.

Newest test packages using Python 3 only

  • irc
  • pydocstyle (also used by other packages)
  • pytest (from version 5.0)
  • Pillow (from Version 7.0)

packages needed for Python 2.7 compatibility

  • ipaddr
  • mock
  • pathlib2
  • requests[security] (for 2.7.4 - 2.7.8)
  • six
  • unicodecsv

[1] https://pythonclock.org/
[2] https://www.python.org/dev/peps/pep-0373/
[3] https://python3statement.org/
[4] https://pypi.org/project/pydocstyle/#description, T215874
[5] https://www.franzoni.eu/python-requests-ssl-and-insecureplatformwarning/

Event Timeline

Xqt created this task.Jan 9 2019, 2:37 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptJan 9 2019, 2:37 PM
Dvorapa added a subscriber: Dvorapa.Jan 9 2019, 3:11 PM
Xqt updated the task description. (Show Details)Jan 9 2019, 3:47 PM
Xqt triaged this task as Lowest priority.Jan 11 2019, 8:25 AM
Dalba awarded a token.Jan 14 2019, 5:56 AM
Xqt updated the task description. (Show Details)Jan 15 2019, 12:05 PM

What about py3.4? It's EOL is planned to 3/2019

Xqt added a comment.Jan 15 2019, 1:41 PM

What about py3.4? It's EOL is planned to 3/2019

Sure but we should drop py2 first which is hard enough ;)
I fear we will get some compatibility problems with py3.4 and py3.5 when py3.8 is released at the end of this year and guess what py 2.7 does.

What about py3.4? It's EOL is planned to 3/2019

Sure but we should drop py2 first which is hard enough ;)
I fear we will get some compatibility problems with py3.4 and py3.5 when py3.8 is released at the end of this year and guess what py 2.7 does.

Yeah, seems logic. Do we have now some warning, that py2.7 will be dropped in a year?

BTW what blocks dropping of py2.7.3 and py3.7 testing?

Xqt added a comment.Jan 15 2019, 2:02 PM

BTW what blocks dropping of py2.7.3 and py3.7 testing?

I made the patch. feel free to merge it.
But T213518 should be merged first to enable test for Travis and Appveyor and solve the known test bug.

Xqt updated the task description. (Show Details)Feb 12 2019, 10:50 AM
Xqt updated the task description. (Show Details)
Xqt added a comment.EditedFeb 12 2019, 11:09 AM

Yeah, seems logic. Do we have now some warning, that py2.7 will be dropped in a year?

Our framework has warning when using it with python <= 2.7.3:

warn(
    'Pywikibot will soon drop support for Python 2.7.2 and 2.7.3, '
    'please update your Python.',
    FutureWarning,

There are some warnings on tests:

and more and more test packages doesn't support Python 2 anymore (pythest is dropping 3.4 with 5.0.0 soon)

Xqt updated the task description. (Show Details)Feb 13 2019, 3:28 PM
Xqt changed the status of subtask T203471: Drop support for Python 2.7.6 and lower from Stalled to Open.
Dvorapa added a comment.EditedFeb 13 2019, 4:41 PM

We could make a plan Pywikibot users could rely on. Like dropping 2.7.6 in 2 months, 2.7.9 in 4 months, some space for another potential deprecation, 2.7 in 8 months, 3.4 in 10 months (just an idea, not an actual proposal).

Xqt updated the task description. (Show Details)Feb 18 2019, 3:50 PM
Xqt updated the task description. (Show Details)Feb 18 2019, 4:05 PM
Xqt updated the task description. (Show Details)Apr 4 2019, 1:54 PM

Change 508093 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] Show deprecation warning for Python 2

https://gerrit.wikimedia.org/r/508093

Change 508093 merged by jenkins-bot:
[pywikibot/core@master] Show deprecation warning for Python 2

https://gerrit.wikimedia.org/r/508093

Dvorapa added a comment.EditedMay 19 2019, 12:31 PM

At Pywikibot meetup there were concerns about the way how to deprecate Python 2 (more details). There are several approaches and many participants would like to not break the current py2 scripts as long as we can (perhaps until major dependencies like pip or requests will break on Python 2)

I like the proposal of archiving Python 2-3 compatible 3.0 master branch and fork it to new 4.0 master branch. But I'm concerned if this would not be a second nightmare like Core-Compat was in the end

Xqt added a comment.May 19 2019, 1:20 PM

I like the proposal of archiving Python 2-3 compatible 3.0 master branch and fork it to new 4.0 master branch. But I'm concerned if this would not be a second nightmare like Core-Compat was

I don't see the same nightmare comming. The problem of compat/core developing was that core started 2007 but it wasn't ready to use. compat was under active development until end of 2016. I would say core was usable from more or less 2012 onwards. Development changed from compat to core in 2012/2013. These 10 years was a very long time of parallel support which is really hard and there are some remaining issues of incompatibility. But I don't see that this will repeated from core to a say pwb 4.0. Anyway we had another kind of parallel development as we had the core/master branch for development and 2.0 branch as stable release between 2014 and 2017 until we dropped 2.0 because backporting code determined as "stable" from master to 2.0 was not maintainable.

Currenly I feel that having Python 2.7 support for a long time prevents or slows down further development and progression because we cannot use new features. I think future bots will working more with streams instead of collecting data. We will have workers for each type of event instead if single bots. I think we should be able using suitable tools for that; this might be means desupporting 2.7 anytime. I don't think this is in next few months.

But in meantime we can write Python 3 code and force operators to install those library parts necessary to use that code instead of installing six on Python 3 environments and prevent using new Python 3 features. Make Pathlib or enum packages mandatory for Python 2.7 for example. This ensures the code to be compatible for a while without slow down develoment too strong. Anyway it is hard enough to test each code for both versions.

Dvorapa added a comment.EditedMay 19 2019, 1:29 PM

Okay, perhaps we can do it this way:

  • archive current master
  • fork a new master from it
  • maintain the old master only to make it work, do not invest more time in it: no backports, no new features, not complicated bugfixes, no code redesigns or other stuff, just simple bugfixes and workarounds for incompatibilities
  • deprecate py2 in the new master completely (remove py2 specific code)
  • encourage people to move to the new master
  • drop the old master at the point pip/requests will drop and break it completely
Xqt added a comment.May 19 2019, 5:20 PM

I propose to use a new "dev" or "4.0" branch for it for the first step. This prevents master users from breaking changes. We can adjust branches and tags later then.

And I wish to implement semver for the new branch. Perhaps we should create a new task to collect tasks to be part of the new branch an not backported to the old.

Dvorapa added a comment.EditedMay 19 2019, 5:24 PM

Implementing semver would require to change our deployment model to PyPI, or not? (also I like the current state)
Also, I don't like backporting. It's time-consuming. We should avoid it.

Xqt added a comment.May 19 2019, 6:00 PM

Implementing semver would require to change our deployment model to PyPI, or not?

No it does not.

KTC added a subscriber: KTC.May 25 2019, 11:00 AM
Eatcha added a subscriber: Eatcha.EditedMay 25 2019, 11:54 AM
WARNING: /shared/pywikipedia/core/pywikibot/__init__.py:132: FutureWarning: Python 2.7.13 will be dropped in 2020. It is recommended to use Python 3.5 or above. See T213287 for further information.
FutureWarning)

Why am I still getting the warning after porting to Python 3 ? Is it just shown irrespective of the python version ?

Please see the following comment of KTC for more info

KTC added a comment.May 25 2019, 12:00 PM

For information, this is the code, and this is the command that ran the script.

Dalba added a subscriber: Dalba.EditedMay 25 2019, 12:05 PM

For information, this is the code, and this is the command that ran the script.

I think you should use python3 in your command, python refers to Python 2 by default.

 0 5,13,21 * * * jsub -once python3 fpc.py -park -close -auto

$ python --version
Python 2.7.13
$ python3 --version
Python 3.5.3

For information, this is the code, and this is the command that ran the script.

I think you should use python3 in your command, python refers to Python 2 by default.

 0 5,13,21 * * * jsub -once python3 fpc.py -park -close -auto
 
$ python --version
Python 2.7.13
$ python3 --version
Python 3.5.3

Thanks for help

Xqt added a comment.Mon, Jun 24, 4:06 PM

Some ideas for dopping Python 2.7:

  • pwb version issue
    • The next pywikibot release number can be 4.0:
      • 1.0 for compat
      • 2.0 for stable release branch of core which has been dropped 2017
      • 3.0 for the core master branch
      • 4.0 for the new branch supporting Python >= 3.X only
  • Python release issues
    • Python 3.4 reached its end-of-life time in March (PEP 429) and should not be supported any longer (1)
    • Python 3.5 reaches its end-of-life time in September 2020 (2)
    • requests library is a base libreary of pwb but requests III needs Python 3.6 (3)
  • Repository issues
    • hard break of core
    • we can either have a new repository for the new and set the curren core repository to be read-only for further use
    • use a new branch for the new (as we had it with core "2.0") and merge new pathes to it. But this can be problematic if we want to change old master with 3.0 and 4.0 with new master at any time.
  • Code issues
    • open patches should be merged as much as possible instead of rebase it to the new repository/branch
    • new features are reserved for the new repository/branch
    • code redesing should be done in the new repository/branch
    • bugfixes can be done for a longer time if necessary or desirable for the old repository/branch

see also T221801

I like the idea of making the current master a read-only repository/branch and start a new one. This way the development will be concentrated on one branch only, and there will be no breakage at all. Or if not read-only, then only merge important bug-fixes, but nothing else. The version should be moved to 4.0 surely (the suggested move to SemVer can be done afterwards, no need to block py2 deprecation on this).

The new branch should be Python 3.5-3.7 as Python 3.5 is currently the main Python on Wikimedia Toolforge/Cloud services/PAWS. We can deprecate Python 3.5 after it will be discontinued on Cloud services too, but I expect it not to be so soon.