requests library breaks several scripts. The scripts hangs after few hours. CPU time consuming is 0%.
The Problem was already described in T91236 for redirect.py and "httplib2". But that bug occured after several weeks with the former library. But now with "requests" I got this bug for other scripts too and it occured after several hours every day.
See also T101228.
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| [bugfix] Timeout session.requests | pywikibot/core | master | +7 -4 |
Related Objects
Related Objects
- Mentioned In
- T158943: Implement EventStreams client for pywikibot
rPWBCd36788195fc0: [bugfix] Timeout session.requests
T72936: Important tasks to be solved (tracking)
T91236: redirect.py hangs - Mentioned Here
- T102113: pywikibot freeze completely
T91236: redirect.py hangs
T101228: [BREAKING] Change: Switch from httplib2 to requests
Event Timeline
Comment Actions
I haven't checked our code for details, but based on the lack of response to ctrl-c and the 0% cpu use, I'm guessing this is a blocking wait() without timeout. Googling shows that requests doesn't set a timeout by default, so that's consistent. It can be set using
http://docs.python-requests.org/en/latest/user/quickstart/#timeouts
Comment Actions
Is this on tool labs, and is one of these processes still running? Then I can take a look to check in detail.
Comment Actions
And, looking at our code, we indeed do not set a timeout: https://github.com/wikimedia/pywikibot-core/blob/master/pywikibot/comms/http.py#L249
Even if that's not the root cause, setting it sounds like a sane plan.
Comment Actions
My bot isn't running on labs. I'll have a look to this problem again and try the timeout. Thanks a lot Merlijn for this hint.
Comment Actions
Change 218616 had a related patch set uploaded (by Xqt):
[bugfix] Timeout session.requests