Convert reflinks to requests
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jayvdb
	Sep 3 2015, 1:59 AM

Description

reflinks.py is one of the few scripts still using urllib2. It should be possible to achieve the same functionality using [[http://docs.python-requests.org/en/latest/|requests]].

Before attempting to switch to requests, the current usage of urllib2 should be analysed to map it to requests functionality (assuming requests has the same functionality -- it might not!).
e.g. reflinks currently only fetches 1000000 bytes from each webpage. requests does have the ability to read only a small amount of a webpage. (to be investigated further)

Details

	Subject	Repo	Branch	Lines +/-
	Set user-agent and convert reflinks.py to use requests	pywikibot/core	master	+77 -53

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Xqt	T68102 use one library for all http requests
Resolved		MtDu	T111300 Convert reflinks to requests
Open	Feature	None	T124007 FTP support for reflinks.py
Open		None	T124138 reflinks should fetch only 1Mb of each linked document

Event Timeline

jayvdb created this task.Sep 3 2015, 1:59 AM

jayvdb claimed this task.

jayvdb raised the priority of this task from to High.

jayvdb updated the task description. (Show Details)

jayvdb added projects: Pywikibot, Pywikibot-network.

jayvdb added subscribers: jayvdb, pywikibot-bugs-list, Beta16.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 3 2015, 1:59 AM

jayvdb added a project: Pywikibot-Scripts.Sep 3 2015, 2:00 AM

jayvdb moved this task from Backlog to references on the Pywikibot-Scripts board.

Mpaa subscribed.Sep 3 2015, 5:59 PM

Liuxinyu970226 set Security to None.Sep 27 2015, 12:44 AM

Mpaa unsubscribed.Sep 27 2015, 11:02 AM

jayvdb mentioned this in T113596: reflinks.py work with old user agent.Oct 16 2015, 7:14 AM

Tokencolour subscribed.Oct 27 2015, 12:35 PM

jayvdb mentioned this in T118674: Pywikibot reflinks.py doesn't work on Python 3 because of Unicode decoding error.Jan 10 2016, 9:56 PM

Rubin16 subscribed.Jan 11 2016, 3:46 AM

I havent looked closely, but I expect that the requests auto-decoding will not have all the same features as the existing reflinks.py decoding. more analysis needed.

On https://gerrit.wikimedia.org/r/#/c/264251/11/, @MtDu wrote:

I looked around and found this. https://toolbelt.readthedocs.org/en/latest/downloadutils.html But when I did that, it said no module named requests_toolbelt. I need to do this for each of the three lines I commented on, or is it different for each? I need to sleep though, so leave me a message here or on IRC. Thanks, MtDu

toolbelt is another package that uses requests, simplifying some tasks. I have quickly looked at its stream capability, and I believe that it will not simplify the reflinks code.

I am also worried that it adds another dependency, and it has a minimum version requirement of requests 2.0.1 (mentioned at https://toolbelt.readthedocs.org/en/latest/user.html)