Maniphest T301908

Drop support for Python 3.5
Closed, ResolvedPublicRelease
Actions

Assigned To

Authored By

	Xqt
	Feb 16 2022, 7:30 PM

Description

Proposal
I propose to drop support for Python 3.5 with Pywikibot 8.0 (deployed probably in September 2022).

Toolforge is going to have Python 3.7 and 3.9 available in May (T268438) and there are further advantages with Python 3.6+.

Please add your remarks, expand pros and cons and maybe draw attention to it when you find a blocker.

Pros

Python 3.6 comes with improvements like path-like objects, f-strings, variable type hints, which are very important for furher Pywikibot developent. [1]
Pywikibot gui cannot be used with Python 3.5 due to T278743
Also a lot of packages does not support Python 3.5 with their current release. This concerns in particular requests package and wikitext parser modules wikitextparser and mwparserfromhell [2]
The end of life cycle of Python 3.5 has been reached since 13th September 2020 [3]
Our documenation recommends Python 3.6 or higher [4]
Python 3.5 does not support pathlike objects. Path from pathlib is often used in Pywikibot and some workarounds are needed to support Python 3.5
In-line re-flags aren’t available with Python 3.5, see T305226
Python 3.5 does not hashlib.blake2b which is used by Pywikibot [5]
Older Python versions are still supported with Pywikibot releases marked with a tag [6][7]
L10N updates can be done by maintenance scripts for older Python versions
Pywikibot 3.5 is not used very often (~ 0.3 %), see T266984
A lot of workarounds are made for Python 3.5 [8]
Python 3.5 does not solve circular import as it is done by newer releases (T30676)
Note: Python 3.6 is the minimal requirement to add entry points of scripts packages to be runned with Pywikibot's pwb wrapper script (T139143)

Cons

Python 3.6 has syntax changes which cannot be backported in any way.
Pywikibot is not updated for older Python versions
Bugfixes are not available for older Pywikibot versions

Refs
[1] https://docs.python.org/3.6/whatsnew/3.6.html
[2] https://codesearch.wmcloud.org/pywikibot/?q=%5B%3C%3D%3E%5D%20%5B'%22%5D3%5C.6&i=nope&files=&excludeFiles=&repos=
[3] https://endoflife.date/python
[4] https://doc.wikimedia.org/pywikibot/stable/index.html
[5] https://docs.python.org/3.11/library/hashlib.html?highlight=blake2b#hashlib.blake2b
[6] https://pypi.org/project/pywikibot/#history
[7] https://gerrit.wikimedia.org/r/admin/repos/pywikibot/core,tags
[8] https://codesearch.wmcloud.org/pywikibot/?q=3(%5C.%7C%2C%20)%5B56%5D%5B%5E%2C%5Cd%5D&i=nope&files=&excludeFiles=&repos=

Details

Subject	Repo	Branch	Lines +/-
Zuul: [pywikibot/core] Use Python 3.6 instead of 3.5 for fasttest	integration/config	master	+3 -3
[PWB8] Drop Python 3.5 dependencies	pywikibot/core	master	+41 -468
[tests] Drop Python 3.5 tests on appveyor	pywikibot/core	master	+6 -14
[8.0] Remove Python 3.5 CI tests on github	pywikibot/core	master	+4 -4
[py35] Show a warning if Pywikibot is running with Python 3.5 and below	pywikibot/core	master	+14 -1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	BUG REPORT	Xqt	T306760 Python 3.5 is not able to solve circular imports like Python 3.6+ do
Open	Feature	None	T57889 Improve support for asynchronous requests (saving/preloading pages)
Resolved	Release	Xqt	T308258 Pywikibot 8 tasks (Tracking)
Resolved	Release	Xqt	T301908 Drop support for Python 3.5
Resolved		Xqt	T305292 migrate Pywikibot from Toolforge Stretch Grid Engine to Toolforge Kubernetes
Resolved		Urbanecm	T312794 Provide some Pywikibot usage statistics for Python3.5 and 3.6

Event Timeline

Xqt created this task.Feb 16 2022, 7:30 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptFeb 16 2022, 7:30 PM

I also suggest dropping support for Python 3.6. Both are EOL.

If we keep 3.6, I suggest only 3.6.1+.

Xqt added a subscriber: Strainu.Feb 17 2022, 12:57 AM

Xqt updated the task description. (Show Details)Feb 17 2022, 12:59 AM

DannyS712 unsubscribed.Feb 17 2022, 2:37 AM

At least bullets 2, 3, 4, and 10 are convincing.

If we keep 3.6, I suggest only 3.6.1+.

Why so?

• taavi subscribed.Feb 17 2022, 8:35 AM

Xqt added a comment.Feb 17 2022, 4:16 PM

This comment was removed by Xqt.

In T301908#7716066, @JJMC89 wrote:

I also suggest dropping support for Python 3.6. Both are EOL.

We still have ~6% who uses Python 3.6, see T266984. Maybe this is much less in few months.

Strainu awarded a token.Feb 24 2022, 11:08 AM

I continue to believe we need to support a version at least a year after Cloud Services have moved away from it, as described in https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/BSTDB6JYJ74DE3BTNWND4BAROLIXJTW5/

That would mean mid-2023 if I understood correctly the latest news about the Debian upgrades.

In T301908#7734698, @Strainu wrote:

I continue to believe we need to support a version at least a year after Cloud Services have moved away from it, as described in https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/BSTDB6JYJ74DE3BTNWND4BAROLIXJTW5/

That would mean mid-2023 if I understood correctly the latest news about the Debian upgrades.

If I follow this your proposal:

two years after official support has ended (e.g. 7 years after launch) AND
one year after Toolforge moved to a newer Python version. AND
the percentage of users of a version goes under 5%

The official support has ended Sept. 2020; two additional years will be Sept 2022. Dropping Python 3.5 is not planed for an earlier date.
Toolforge migration has already started.
The percentage of users is already below 0,3 % except for Toolforge but this will decrease rapidly if the Python base is changed.

I don't want to fight for a few weeks or months. Release Pywikibot 7 was also postponed for few months. But I want to promote a change as early as even possible but this is also a long term process so that it can be planned. Please take into account that other packages that are mandatory for Pywikibot need Python 3.6 already; for example the central http interface requests has dropped Python 3.5 support 7 months ago and Python 3.6 support will be dropped this year. See also https://codesearch.wmcloud.org/pywikibot/?q=3(%5C.%7C%2C%20)%5B56%5D&i=nope&files=&excludeFiles=&repos= to keep Python 3.5 compatibility.

In T301908#7734951, @Xqt wrote:

In T301908#7734698, @Strainu wrote:

I continue to believe we need to support a version at least a year after Cloud Services have moved away from it, as described in https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/BSTDB6JYJ74DE3BTNWND4BAROLIXJTW5/

That would mean mid-2023 if I understood correctly the latest news about the Debian upgrades.

If I follow this your proposal:

two years after official support has ended (e.g. 7 years after launch) AND

one year after Toolforge moved to a newer Python version. AND

the percentage of users of a version goes under 5%

The official support has ended Sept. 2020; two additional years will be Sept 2022. Dropping Python 3.5 is not planed for an earlier date.

Toolforge migration has already started.

The percentage of users is already below 0,3 % except for Toolforge but this will decrease rapidly if the Python base is changed.

I don't want to fight for a few weeks or months. Release Pywikibot 7 was also postponed for few months. But I want to promote a change as early as even possible but this is also a long term process so that it can be planned.

OK, I checked the announcement and it seems that the final shutdown of Stretch is planned to happen around June (although I suspect it will get delayed a little, but probably not until October). Let's get some data after that happens and see if there is anyone still using 3.5 from the Cloud Services (which would mean they explicitly need it) and if not then we can keep your proposed date.

If there are still 3.5 users after June, we can still keep the date, but we should arrange some kind of notification (not sure who and in what circumstances can do that).

How does that sound?

Please take into account that other packages that are mandatory for Pywikibot need Python 3.6 already; for example the central http interface requests has dropped Python 3.5 support 7 months ago and Python 3.6 support will be dropped this year.

It's not immediately obvious to me what's the concern here. What's forcing us to increase the required requests version?

How does that sound?

Yes I agree with this, we are not in any hurry. The proposed date can easily be postponed if necessary to give bot owners time enough to migrate their Python. It could also be December or March 23. A notification is planned after toolforge is migrated. After a while we check the usage like we have done it for T286867.

It's not immediately obvious to me what's the concern here. What's forcing us to increase the required requests version?

The advantages of newer versions are that some bugs are fixed, some security issues are solved and some improvements were made. In many cases this may not matter a lot. Pillow for example only works with Python 3.6+ due to a security issue.

Xqt updated the task description. (Show Details)Feb 28 2022, 3:38 PM

Xqt triaged this task as Lowest priority.Mar 31 2022, 8:11 AM

Xqt added a project: Pywikibot-Announce.

Xqt added a subtask: T305292: migrate Pywikibot from Toolforge Stretch Grid Engine to Toolforge Kubernetes.Apr 2 2022, 9:01 AM

Xqt added subscribers: Ricordisamoa, MarcoAurelio.Apr 4 2022, 4:38 AM

Xqt added a subscriber: Pywikibot.

Xqt added a subscriber: Pywikibot-Announce.Apr 4 2022, 4:43 AM

Xqt updated the task description. (Show Details)Apr 4 2022, 5:07 AM

Aklapper removed subscribers: Pywikibot-Announce, Pywikibot.Apr 16 2022, 6:49 AM

Xqt changed the subtype of this task from "Task" to "Release".Apr 21 2022, 2:41 PM

Xqt mentioned this in T306637: Drop support for MediaWiki < 1.27.Apr 21 2022, 3:49 PM

Xqt raised the priority of this task from Lowest to Medium.Apr 24 2022, 2:19 PM

Xqt added a parent task: T306760: Python 3.5 is not able to solve circular imports like Python 3.6+ do.

Xqt updated the task description. (Show Details)Apr 24 2022, 2:26 PM

Xqt moved this task from Backlog to Next main release on the Pywikibot board.Apr 25 2022, 8:00 AM

Xqt added a subtask: T139143: Make a pywikibot entry point for scripts.Apr 25 2022, 8:06 AM

Xqt updated the task description. (Show Details)

Xqt added a parent task: T57889: Improve support for asynchronous requests (saving/preloading pages).May 12 2022, 10:37 AM

Xqt added a parent task: T308258: Pywikibot 8 tasks (Tracking).May 12 2022, 4:11 PM

Xqt removed a subtask: T139143: Make a pywikibot entry point for scripts.

Change 792930 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [py35] Show a warning if Pywikibot is running with Python 3.5 and below

https://gerrit.wikimedia.org/r/792930

gerritbot added a project: Patch-For-Review.May 18 2022, 5:46 AM

Xqt closed subtask T305292: migrate Pywikibot from Toolforge Stretch Grid Engine to Toolforge Kubernetes as Resolved.May 18 2022, 6:30 AM

Change 792930 merged by jenkins-bot:

[pywikibot/core@master] [py35] Show a warning if Pywikibot is running with Python 3.5 and below

https://gerrit.wikimedia.org/r/792930

Maintenance_bot removed a project: Patch-For-Review.May 18 2022, 5:30 PM

Ibkt updated the task description. (Show Details)Jul 1 2022, 5:26 PM

Xqt removed a project: Pywikibot-RfCs.Jul 1 2022, 5:46 PM

Xqt added a subtask: T312794: Provide some Pywikibot usage statistics for Python3.5 and 3.6.Jul 11 2022, 4:55 PM

Highest requirements due to toolforge installation:

package	version
Python	3.7.3
setuptools	40.8.0
mwparserfromhell	0.6.3
wikitextparser	n/a
requests	2.21.0
pymysql	0.9.3
sseclient	n/a
python-stdnum	n/a
pydot	1.4.1
google	n/a
memento_client	n/a
tkinter	?
PIL	5.4.1
mwoauth	n/a
bs4	4.7.1
fake_useragent	n/a
isbnlib	n/a