Page MenuHomePhabricator

Drop support for Python 3.5
Open, MediumPublicRelease

Description

Proposal
I propose to drop support for Python 3.5 with Pywikibot 8.0 (deployed probably in September 2022).

Toolforge is going to have Python 3.7 and 3.9 available in May (T268438) and there are further advantages with Python 3.6+.

Please add your remarks, expand pros and cons and maybe draw attention to it when you find a blocker.

Pros

  • Python 3.6 comes with improvements like path-like objects, f-strings, variable type hints, which are very important for furher Pywikibot developent. [1]
  • Pywikibot gui cannot be used with Python 3.5 due to T278743
  • Also a lot of packages does not support Python 3.5 with their current release. This concerns in particular requests package and wikitext parser modules wikitextparser and mwparserfromhell [2]
  • The end of life cycle of Python 3.5 has been reached since 13th September 2020 [3]
  • Our documenation recommends Python 3.6 or higher [4]
  • Python 3.5 does not support pathlike objects. Path from pathlib is often used in Pywikibot and some workarounds are needed to support Python 3.5
  • In-line re-flags aren’t available with Python 3.5, see T305226
  • Python 3.5 does not hashlib.blake2b which is used by Pywikibot [5]
  • Older Python versions are still supported with Pywikibot releases marked with a tag [6][7]
  • L10N updates can be done by maintenance scripts for older Python versions
  • Pywikibot 3.5 is not used very often (~ 0.3 %), see T266984
  • A lot of workarounds are made for Python 3.5 [8]
  • Python 3.5 does not solve circular import as it is done by newer releases (T30676)
  • Note: Python 3.6 is the minimal requirement to add entry points of scripts packages to be runned with Pywikibot's pwb wrapper script (T139143)

Cons

  • Python 3.6 has syntax changes which cannot be backported in any way.
  • Pywikibot is not updated for older Python versions
  • Bugfixes are not available for older Pywikibot versions

Refs
[1] https://docs.python.org/3.6/whatsnew/3.6.html
[2] https://codesearch.wmcloud.org/pywikibot/?q=%5B%3C%3D%3E%5D%20%5B'%22%5D3%5C.6&i=nope&files=&excludeFiles=&repos=
[3] https://endoflife.date/python
[4] https://doc.wikimedia.org/pywikibot/stable/index.html
[5] https://docs.python.org/3.11/library/hashlib.html?highlight=blake2b#hashlib.blake2b
[6] https://pypi.org/project/pywikibot/#history
[7] https://gerrit.wikimedia.org/r/admin/repos/pywikibot/core,tags
[8] https://codesearch.wmcloud.org/pywikibot/?q=3(%5C.%7C%2C%20)%5B56%5D%5B%5E%2C%5Cd%5D&i=nope&files=&excludeFiles=&repos=

Event Timeline

I also suggest dropping support for Python 3.6. Both are EOL.

If we keep 3.6, I suggest only 3.6.1+.

At least bullets 2, 3, 4, and 10 are convincing.

If we keep 3.6, I suggest only 3.6.1+.

Why so?

This comment was removed by Xqt.

I also suggest dropping support for Python 3.6. Both are EOL.

We still have ~6% who uses Python 3.6, see T266984. Maybe this is much less in few months.

I continue to believe we need to support a version at least a year after Cloud Services have moved away from it, as described in https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/BSTDB6JYJ74DE3BTNWND4BAROLIXJTW5/

That would mean mid-2023 if I understood correctly the latest news about the Debian upgrades.

I continue to believe we need to support a version at least a year after Cloud Services have moved away from it, as described in https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/BSTDB6JYJ74DE3BTNWND4BAROLIXJTW5/

That would mean mid-2023 if I understood correctly the latest news about the Debian upgrades.

If I follow this your proposal:

  1. two years after official support has ended (e.g. 7 years after launch) AND
  2. one year after Toolforge moved to a newer Python version. AND
  3. the percentage of users of a version goes under 5%
  1. The official support has ended Sept. 2020; two additional years will be Sept 2022. Dropping Python 3.5 is not planed for an earlier date.
  2. Toolforge migration has already started.
  3. The percentage of users is already below 0,3 % except for Toolforge but this will decrease rapidly if the Python base is changed.

I don't want to fight for a few weeks or months. Release Pywikibot 7 was also postponed for few months. But I want to promote a change as early as even possible but this is also a long term process so that it can be planned. Please take into account that other packages that are mandatory for Pywikibot need Python 3.6 already; for example the central http interface requests has dropped Python 3.5 support 7 months ago and Python 3.6 support will be dropped this year. See also https://codesearch.wmcloud.org/pywikibot/?q=3(%5C.%7C%2C%20)%5B56%5D&i=nope&files=&excludeFiles=&repos= to keep Python 3.5 compatibility.

I continue to believe we need to support a version at least a year after Cloud Services have moved away from it, as described in https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/BSTDB6JYJ74DE3BTNWND4BAROLIXJTW5/

That would mean mid-2023 if I understood correctly the latest news about the Debian upgrades.

If I follow this your proposal:

  1. two years after official support has ended (e.g. 7 years after launch) AND
  2. one year after Toolforge moved to a newer Python version. AND
  3. the percentage of users of a version goes under 5%
  1. The official support has ended Sept. 2020; two additional years will be Sept 2022. Dropping Python 3.5 is not planed for an earlier date.
  2. Toolforge migration has already started.
  3. The percentage of users is already below 0,3 % except for Toolforge but this will decrease rapidly if the Python base is changed.

I don't want to fight for a few weeks or months. Release Pywikibot 7 was also postponed for few months. But I want to promote a change as early as even possible but this is also a long term process so that it can be planned.

OK, I checked the announcement and it seems that the final shutdown of Stretch is planned to happen around June (although I suspect it will get delayed a little, but probably not until October). Let's get some data after that happens and see if there is anyone still using 3.5 from the Cloud Services (which would mean they explicitly need it) and if not then we can keep your proposed date.

If there are still 3.5 users after June, we can still keep the date, but we should arrange some kind of notification (not sure who and in what circumstances can do that).

How does that sound?

Please take into account that other packages that are mandatory for Pywikibot need Python 3.6 already; for example the central http interface requests has dropped Python 3.5 support 7 months ago and Python 3.6 support will be dropped this year.

It's not immediately obvious to me what's the concern here. What's forcing us to increase the required requests version?

How does that sound?

Yes I agree with this, we are not in any hurry. The proposed date can easily be postponed if necessary to give bot owners time enough to migrate their Python. It could also be December or March 23. A notification is planned after toolforge is migrated. After a while we check the usage like we have done it for T286867.

It's not immediately obvious to me what's the concern here. What's forcing us to increase the required requests version?

The advantages of newer versions are that some bugs are fixed, some security issues are solved and some improvements were made. In many cases this may not matter a lot. Pillow for example only works with Python 3.6+ due to a security issue.

Xqt triaged this task as Lowest priority.Mar 31 2022, 8:11 AM
Xqt added a project: Pywikibot-Announce.
Xqt changed the subtype of this task from "Task" to "Release".Apr 21 2022, 2:41 PM

Change 792930 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [py35] Show a warning if Pywikibot is running with Python 3.5 and below

https://gerrit.wikimedia.org/r/792930

Change 792930 merged by jenkins-bot:

[pywikibot/core@master] [py35] Show a warning if Pywikibot is running with Python 3.5 and below

https://gerrit.wikimedia.org/r/792930