Page MenuHomePhabricator

Provide some pywikibot usage statistics for Python3.6 and 3.7
Closed, ResolvedPublic

Description

Python 3.6. as well as Python 3.7 is EOL. For Pywikibot it is planed to drop Python 3.6 support soon but we should have a data base how often outdated Python releases are still used. It would be useful to have some statistics for that.

See older requests made in T242157, T286867 T312794 for previous release changes.

Event Timeline

mevo removed a subscriber: mevo.

@Legoktm, @Urbanecm: Could one of you take over this task. Looks like it is too complicated to make it by my own.

@Legoktm, @Urbanecm: Could one of you take over this task. Looks like it is too complicated to make it by my own.

I think the last step for you to get access to the data yourself is answering the T348520#9335272 question at the NDA task (and confirming the e-mail address you have currently set in LDAP). As far as I can see, all the relevant approvals are already recorded on the task. Sorry it is a complicated process – fortunately, it's something that only needs to be used once.

If there's actually more things to complete on the NDA task that I'm not seeing, I can take a look at generating the stats for you myself, but if the NDA access for you is as close as I think it is, then I'd prefer you answering that question and getting access to it yourself (if needed, I'd be more than happy to show you around, but the interface is fairly intuitive). Let me know.

@Urbanecm: Thank you. Currently I have no glue how can I get access to stat1005 and how to call the python statements you did last year in T312794. Can you lead me for the first steps?

@Urbanecm: Thank you. Currently I have no glue how can I get access to stat1005 and how to call the python statements you did last year in T312794. Can you lead me for the first steps?

stat1005 is not accessible to you at this point, but having access to the stat servers is not strictly needed to get a reasonable usage overview. Since the NDA request is now granted, you should be able to follow my steps from T242157#5785171, where I used the webrequest_sampled_128 dataset (1/128 of all Wikimedia webrequests) in https://turnilo.wikimedia.org/. While it will not give you data about all of the users, because of the sampling and because Turnilo only shows a limited amount of top user agents, I think for deprecation purposes it should be sufficient.

Thanks a lot, I will try that.

Here are some statistic data fro Pywikibot usage [%]:

Pywikibot876543Sum
ALL70111415100
Python 3.111212
Python 3.10103315
Python 3.9105924
Python 3.8101112
Python 3.72211126
Python 3.64116
Python 2.755
Xqt closed this task as Resolved.EditedDec 5 2023, 2:02 PM
Xqt claimed this task.

Conclusion:

There are still 4% using Python 3.6 with Pywikibot 8. Most of them are made by the https://www.marefa.org project reading data from commons (I guess) by using Pywikibot 8.4. Excluding these 200k requests the remaining usage for Python 3.6 with Pywikibot 8 is 0.6% only, most of them is caused by Pywikibot testing.

I haven't found any usage for Python 3.6 with Pywikibot 8.5 and 8.6 except Pywikibot-test requests. Therefore dropping Python 3.6 can be done now safely in pariculary because older Pywikibot releases aren't affected with such change.

@Urbanecm: I wasn't able to exclude Pywikibot-test from User Agent regex Pywikibot/8\.[56].+Python/3\.6 e.g. with (?!test) but get a 504 error. Any idea?

@Urbanecm: I wasn't able to exclude Pywikibot-test from User Agent regex Pywikibot/8\.[56].+Python/3\.6 e.g. with (?!test) but get a 504 error. Any idea?

Besides filtering those agents out during post-processing, not really. 504 indicates a timeout, which makes sense, given negative lookahead is not exactly a cheap query. Granted, removing those agents during postprocessing means you lose certain UAs, but since Turnilo currently supports up to 10k of top user agents, and the least frequent UAs have 1 hit per last month, it should be good enough.