Page MenuHomePhabricator

Drop support of python 2.7
Open, LowestPublic

Description

Python 2.7 as reached his EOL release and will not be maintained after January 1st, 2020 [1] due to PEP 373 [2]. More and more projects withdraw support of python 2.7 [3] including test packages [4]

I propose to use the full potental for python 3 and to cleanup code from old stuff. This hasn't to be immediately but bot owners should be conducted to Switch so python 3 soon.

What can be the steps to proceed?

  • Abandon support for Python 2.6 (T154771)
  • Drop support for Python 3.3 (T184508)
  • Drop support for python 2.7.2 and 2.7.3 now (T191192)
  • Drop support for Python 2.7.6 and lower in few months (T203471)
  • Drop support for Python 2.7.8 and lower in few months due to InsecureWarning of urllib3 [5]
  • Withdraw Python 2.7.2/2.7.3 from appveyor test
  • Add python 3.7 to test matrix gerrit:433554
  • Add python 3.8 to test matrix gerrit:482532
  • New functionalities for python 3 only
  • Drop support for python 2.7 in 2020

If someone cannot upgrade to Python 3.0, the older Pywikibot releases are still available either via pypi package or the corresponding tag in our repository but one should be aware that these issues may still lead to problems.

Newest test packages using Python 3 only

  • irc
  • pydocstyle (also used by other packages)
  • pytest (from version 5.0)
  • Pillow (from Version 7.0)

packages needed for Python 2.7 compatibility

  • ipaddr
  • mock
  • pathlib2
  • requests[security] (for 2.7.4 - 2.7.8)
  • six
  • unicodecsv

[1] https://pythonclock.org/
[2] https://www.python.org/dev/peps/pep-0373/
[3] https://python3statement.org/
[4] https://pypi.org/project/pydocstyle/#description, T215874
[5] https://www.franzoni.eu/python-requests-ssl-and-insecureplatformwarning/

Related Objects

Event Timeline

Xqt created this task.Jan 9 2019, 2:37 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptJan 9 2019, 2:37 PM
Dvorapa added a subscriber: Dvorapa.Jan 9 2019, 3:11 PM
Xqt updated the task description. (Show Details)Jan 9 2019, 3:47 PM
Xqt triaged this task as Lowest priority.Jan 11 2019, 8:25 AM
Dalba awarded a token.Jan 14 2019, 5:56 AM
Xqt updated the task description. (Show Details)Jan 15 2019, 12:05 PM

What about py3.4? It's EOL is planned to 3/2019

Xqt added a comment.Jan 15 2019, 1:41 PM

What about py3.4? It's EOL is planned to 3/2019

Sure but we should drop py2 first which is hard enough ;)
I fear we will get some compatibility problems with py3.4 and py3.5 when py3.8 is released at the end of this year and guess what py 2.7 does.

What about py3.4? It's EOL is planned to 3/2019

Sure but we should drop py2 first which is hard enough ;)
I fear we will get some compatibility problems with py3.4 and py3.5 when py3.8 is released at the end of this year and guess what py 2.7 does.

Yeah, seems logic. Do we have now some warning, that py2.7 will be dropped in a year?

BTW what blocks dropping of py2.7.3 and py3.7 testing?

Xqt added a comment.Jan 15 2019, 2:02 PM

BTW what blocks dropping of py2.7.3 and py3.7 testing?

I made the patch. feel free to merge it.
But T213518 should be merged first to enable test for Travis and Appveyor and solve the known test bug.

Xqt updated the task description. (Show Details)Feb 12 2019, 10:50 AM
Xqt updated the task description. (Show Details)
Xqt added a comment.EditedFeb 12 2019, 11:09 AM

Yeah, seems logic. Do we have now some warning, that py2.7 will be dropped in a year?

Our framework has warning when using it with python <= 2.7.3:

warn(
    'Pywikibot will soon drop support for Python 2.7.2 and 2.7.3, '
    'please update your Python.',
    FutureWarning,

There are some warnings on tests:

and more and more test packages doesn't support Python 2 anymore (pythest is dropping 3.4 with 5.0.0 soon)

Xqt updated the task description. (Show Details)Feb 13 2019, 3:28 PM
Xqt changed the status of subtask T203471: Drop support for Python 2.7.6 and lower from Stalled to Open.
Dvorapa added a comment.EditedFeb 13 2019, 4:41 PM

We could make a plan Pywikibot users could rely on. Like dropping 2.7.6 in 2 months, 2.7.9 in 4 months, some space for another potential deprecation, 2.7 in 8 months, 3.4 in 10 months (just an idea, not an actual proposal).

Xqt updated the task description. (Show Details)Feb 18 2019, 3:50 PM
Xqt updated the task description. (Show Details)Feb 18 2019, 4:05 PM
Xqt updated the task description. (Show Details)Apr 4 2019, 1:54 PM

Change 508093 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] Show deprecation warning for Python 2

https://gerrit.wikimedia.org/r/508093

Change 508093 merged by jenkins-bot:
[pywikibot/core@master] Show deprecation warning for Python 2

https://gerrit.wikimedia.org/r/508093

Dvorapa added a comment.EditedMay 19 2019, 12:31 PM

At Pywikibot meetup there were concerns about the way how to deprecate Python 2 (more details). There are several approaches and many participants would like to not break the current py2 scripts as long as we can (perhaps until major dependencies like pip or requests will break on Python 2)

I like the proposal of archiving Python 2-3 compatible 3.0 master branch and fork it to new 4.0 master branch. But I'm concerned if this would not be a second nightmare like Core-Compat was in the end

Xqt added a comment.May 19 2019, 1:20 PM

I like the proposal of archiving Python 2-3 compatible 3.0 master branch and fork it to new 4.0 master branch. But I'm concerned if this would not be a second nightmare like Core-Compat was

I don't see the same nightmare comming. The problem of compat/core developing was that core started 2007 but it wasn't ready to use. compat was under active development until end of 2016. I would say core was usable from more or less 2012 onwards. Development changed from compat to core in 2012/2013. These 10 years was a very long time of parallel support which is really hard and there are some remaining issues of incompatibility. But I don't see that this will repeated from core to a say pwb 4.0. Anyway we had another kind of parallel development as we had the core/master branch for development and 2.0 branch as stable release between 2014 and 2017 until we dropped 2.0 because backporting code determined as "stable" from master to 2.0 was not maintainable.

Currenly I feel that having Python 2.7 support for a long time prevents or slows down further development and progression because we cannot use new features. I think future bots will working more with streams instead of collecting data. We will have workers for each type of event instead if single bots. I think we should be able using suitable tools for that; this might be means desupporting 2.7 anytime. I don't think this is in next few months.

But in meantime we can write Python 3 code and force operators to install those library parts necessary to use that code instead of installing six on Python 3 environments and prevent using new Python 3 features. Make Pathlib or enum packages mandatory for Python 2.7 for example. This ensures the code to be compatible for a while without slow down develoment too strong. Anyway it is hard enough to test each code for both versions.

Dvorapa added a comment.EditedMay 19 2019, 1:29 PM

Okay, perhaps we can do it this way:

  • archive current master
  • fork a new master from it
  • maintain the old master only to make it work, do not invest more time in it: no backports, no new features, not complicated bugfixes, no code redesigns or other stuff, just simple bugfixes and workarounds for incompatibilities
  • deprecate py2 in the new master completely (remove py2 specific code)
  • encourage people to move to the new master
  • drop the old master at the point pip/requests will drop and break it completely
Xqt added a comment.May 19 2019, 5:20 PM

I propose to use a new "dev" or "4.0" branch for it for the first step. This prevents master users from breaking changes. We can adjust branches and tags later then.

And I wish to implement semver for the new branch. Perhaps we should create a new task to collect tasks to be part of the new branch an not backported to the old.

Dvorapa added a comment.EditedMay 19 2019, 5:24 PM

Implementing semver would require to change our deployment model to PyPI, or not? (also I like the current state)
Also, I don't like backporting. It's time-consuming. We should avoid it.

Xqt added a comment.May 19 2019, 6:00 PM

Implementing semver would require to change our deployment model to PyPI, or not?

No it does not.

KTC added a subscriber: KTC.May 25 2019, 11:00 AM
Eatcha added a subscriber: Eatcha.EditedMay 25 2019, 11:54 AM
WARNING: /shared/pywikipedia/core/pywikibot/__init__.py:132: FutureWarning: Python 2.7.13 will be dropped in 2020. It is recommended to use Python 3.5 or above. See T213287 for further information.
FutureWarning)

Why am I still getting the warning after porting to Python 3 ? Is it just shown irrespective of the python version ?

Please see the following comment of KTC for more info

KTC added a comment.May 25 2019, 12:00 PM

For information, this is the code, and this is the command that ran the script.

Dalba added a subscriber: Dalba.EditedMay 25 2019, 12:05 PM

For information, this is the code, and this is the command that ran the script.

I think you should use python3 in your command, python refers to Python 2 by default.

 0 5,13,21 * * * jsub -once python3 fpc.py -park -close -auto

$ python --version
Python 2.7.13
$ python3 --version
Python 3.5.3

For information, this is the code, and this is the command that ran the script.

I think you should use python3 in your command, python refers to Python 2 by default.

 0 5,13,21 * * * jsub -once python3 fpc.py -park -close -auto
$ python --version
Python 2.7.13
$ python3 --version
Python 3.5.3

Thanks for help

Xqt added a comment.Jun 24 2019, 4:06 PM

Some ideas for dopping Python 2.7:

  • pwb version issue
    • The next pywikibot release number can be 4.0:
      • 1.0 for compat
      • 2.0 for stable release branch of core which has been dropped 2017
      • 3.0 for the core master branch
      • 4.0 for the new branch supporting Python >= 3.X only
  • Python release issues
    • Python 3.4 reached its end-of-life time in March (PEP 429) and should not be supported any longer (1)
    • Python 3.5 reaches its end-of-life time in September 2020 (2)
    • requests library is a base libreary of pwb but requests III needs Python 3.6 (3)
  • Repository issues
    • hard break of core
    • we can either have a new repository for the new and set the curren core repository to be read-only for further use
    • use a new branch for the new (as we had it with core "2.0") and merge new pathes to it. But this can be problematic if we want to change old master with 3.0 and 4.0 with new master at any time.
  • Code issues
    • open patches should be merged as much as possible instead of rebase it to the new repository/branch
    • new features are reserved for the new repository/branch
    • code redesing should be done in the new repository/branch
    • bugfixes can be done for a longer time if necessary or desirable for the old repository/branch

see also T221801

I like the idea of making the current master a read-only repository/branch and start a new one. This way the development will be concentrated on one branch only, and there will be no breakage at all. Or if not read-only, then only merge important bug-fixes, but nothing else. The version should be moved to 4.0 surely (the suggested move to SemVer can be done afterwards, no need to block py2 deprecation on this).

The new branch should be Python 3.5-3.7 as Python 3.5 is currently the main Python on Wikimedia Toolforge/Cloud services/PAWS. We can deprecate Python 3.5 after it will be discontinued on Cloud services too, but I expect it not to be so soon.

Do we have a clear overview of the advantages and disadvantages of dropping 2.7 support? The EOL part is really unimportant to me in this case. It's not like we have a support contract. We have to fix stuff ourselves anyway. I think we can sum it up as:

  • Keeping 2.7 support means more work for the developers of Pywikibot
  • Dropping 2.7 support means more work and breakage for the developers who use Pywikibot as a library

This is like core/compat all over again. We need to balance these groups. Phabricator is mostly populated by developers from the first group. Please take it slow to not alienate the second group. The second group generally builds on top of the core part of Pywikibot. As long as we keep that somewhat functioning, these users will be happy. Scripts on the other hand, you could probably go full Python3 on some point in time.

Xqt added a comment.Jun 30 2019, 5:31 PM

Thanks @Multichill for your comment. Infact keeping 2.7 support means more developing time keeping backward compatible instead of using that time for bringing forward the library with bugfixes new features and don't forget reviewing. We have now 170 open patches ready to submit.

EOL of Python 2.7 is also important for us because the most external libraries are also giving up Python 2 support (including pip btw). This means no further support, no bugfixes no new features. Ok you can say you never update libraries like requests or sseclient and what care about other stuff or about tests. On the other hand you don't need to update your bot. You can use every stable release submitted with their own tag; ask the HISTORY.rst file for their changes. Surely you will find a release running with Python 2.6. But this is not actually tested with current environments and external library dependencies.

Test are also a notable point for some difficult problems, maybe a minor one. Currently complete tests need up to 4 hours to pass; dropping Python 2.7 decreases testing time by 30%.

There are a lot of improvements in Python 3 we cannot use currently like ipaddress, Pathlib or Enum (Enum is the preferred base for a Namespace implementation for example to simplify the code and make it more robust). OK there are libraries which can backport these libaries to Python 2.7. But not doing this also leads to breakage if they are mandatory.

Coroutines are available in Python 2.7 but simplified in Python 3. The addition of the asyncio library makes writing asynchronous programs easier.

Souped unpacking is another advantage and could solve the getVersionHistory bug T136513 (where core implementation is not compatible with core) very easily for example. Currently we have a breaking change when porting compat to core without any hint how to proceed.

Chainging exceptions can help for bug fixes because it gives a more detailes error report.

Python 2 the concept of bytes and strings were pretty much interchangeable, and both came under the str type. In result we have still some UnicodeDecodeError bugs inside the code.

Syntax changes of accessing parent classes and creating classes from metaclasses are heavy in Python 2 but simplier in Python 3.

There are other syntactical changes e.g. related for interators/generators which are different on Python 2 and 3

A lot of buildins uses iterators instead of lists which save significant amounts of memory.

There are still other reasons to upgrade to Python 3 – not just because of the convenience of new features, but because random, obscure bugs in the language where fixed in Python 3 and other libraries depending from it, Pywikibot is not a never-need-to-change codebase because MediaWiki also is under development too. Therefore we should find a way to drop the old and force the new imho.

Sure dropping Python 2.7 should not be made hastily but developers using pywikibot also have their own responsibility and I don't agree that we have to support all deprecated items for the next few decades. There is always the way back to use an older release if the framework development is too fast. But I don't beleave that this is true when remembering that a lot of bugs can be solved immediately.