Page MenuHomePhabricator

[break] request library isn't stable
Closed, ResolvedPublic

Description

requests library breaks several scripts. The scripts hangs after few hours. CPU time consuming is 0%.
The Problem was already described in T91236 for redirect.py and "httplib2". But that bug occured after several weeks with the former library. But now with "requests" I got this bug for other scripts too and it occured after several hours every day.
See also T101228.

Details

Related Gerrit Patches:
pywikibot/core : master[bugfix] Timeout session.requests

Event Timeline

Xqt created this task.Jun 15 2015, 9:19 AM
Xqt raised the priority of this task from to High.
Xqt updated the task description. (Show Details)
Xqt added a project: Pywikibot.
Xqt added a subscriber: Xqt.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptJun 15 2015, 9:19 AM
XZise added a subscriber: XZise.Jun 15 2015, 9:27 AM

Do you get a consistent traceback when you press Ctrl+C or does it get ignored?

Xqt added a comment.Jun 15 2015, 9:28 AM

Ctrl+C doesn't work anymore with that hanging

jayvdb added a subscriber: jayvdb.Jun 15 2015, 9:35 AM
valhallasw added a subscriber: valhallasw.EditedJun 15 2015, 8:51 PM

I haven't checked our code for details, but based on the lack of response to ctrl-c and the 0% cpu use, I'm guessing this is a blocking wait() without timeout. Googling shows that requests doesn't set a timeout by default, so that's consistent. It can be set using

http://docs.python-requests.org/en/latest/user/quickstart/#timeouts

Is this on tool labs, and is one of these processes still running? Then I can take a look to check in detail.

And, looking at our code, we indeed do not set a timeout: https://github.com/wikimedia/pywikibot-core/blob/master/pywikibot/comms/http.py#L249

Even if that's not the root cause, setting it sounds like a sane plan.

Xqt added a comment.Jun 15 2015, 11:01 PM

My bot isn't running on labs. I'll have a look to this problem again and try the timeout. Thanks a lot Merlijn for this hint.

Change 218616 had a related patch set uploaded (by Xqt):
[bugfix] Timeout session.requests

https://gerrit.wikimedia.org/r/218616

Change 218616 merged by jenkins-bot:
[bugfix] Timeout session.requests

https://gerrit.wikimedia.org/r/218616

Xqt closed this task as Resolved.Jun 16 2015, 1:04 PM
Xqt claimed this task.