LiveRCPageGenerator using socketIO_client stall the builds regularly
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jayvdb
	Jan 3 2015, 4:17 AM

Description

LiveRCPageGenerator stall the travis builds. The only common part of the log is

No handlers could be found for logger "socketIO_client"

That message doesnt appear on successful runs of this test. My guess is that "socketIO_client" wants to report a problem (possible socket timeout?), but cant due to a bug in their library or in our use of their library.

ar.wp - 2.6 (only failed build for that commit)
https://travis-ci.org/wikimedia/pywikibot-core/jobs/45687549

ar.wp - 2.7 & 2.6
https://travis-ci.org/wikimedia/pywikibot-core/jobs/45743553
https://travis-ci.org/wikimedia/pywikibot-core/jobs/45743561

It could be triggered by a low amount of data in the rc stream.

Details

	Subject	Repo	Branch	Lines +/-
	RCStream: return heartbeats and handle on_reconnect	pywikibot/core	master	+10 -1
	Prevent hang in LiveRCPageGenerator	pywikibot/core	master	+5 -5

Customize query in gerrit

Related Objects

Mentioned In: rPWBCba4e1e2ef10e: RCStream: return heartbeats and handle on_reconnect
rPWBCe39165a1a99b: Prevent hang in LiveRCPageGenerator
T91394: requests 2.5.3 is causing issues on import
T66828: Add live recent changes page generator
T86506: rcstream doesnt log important status changes within a socketio connection
T85716: LiveRCPageGenerator socketIO_client upstream issues
Mentioned Here: T91393: RCStream is not accessible from python client due to using socket-io 1.0 while only socket-io 0.9 is offered
T86771: RCStream should remember subscriptions after socket.io re-connects

Event Timeline

jayvdb created this task.Jan 3 2015, 4:17 AM

jayvdb raised the priority of this task from to Unbreak Now!.

jayvdb updated the task description. (Show Details)

jayvdb added a project: Pywikibot.

jayvdb added a project: Pywikibot-tests.

jayvdb added subscribers: Aklapper, Unknown Object (MLST), jayvdb.

two more, always ar.wp, and never py3
https://travis-ci.org/wikimedia/pywikibot-core/builds/45747943
https://travis-ci.org/wikimedia/pywikibot-core/builds/45763062

Again ar.wp

https://travis-ci.org/wikimedia/pywikibot-core/jobs/46326040

And another, also ar.wp

https://travis-ci.org/wikimedia/pywikibot-core/jobs/46238478

And another
https://travis-ci.org/jayvdb/pywikibot-core/jobs/46413002

And another:
https://travis-ci.org/wikimedia/pywikibot-core/jobs/46431515
https://travis-ci.org/wikimedia/pywikibot-core/jobs/46431523

jayvdb added a subscriber: valhallasw.Jan 9 2015, 2:34 PM

jayvdb renamed this task from LiveRCPageGenerator stall the builds regularly to LiveRCPageGenerator using socketIO_client stall the builds regularly.Jan 11 2015, 11:50 PM

jayvdb set Security to None.

jayvdb mentioned this in T85716: LiveRCPageGenerator socketIO_client upstream issues.Jan 11 2015, 11:53 PM

more of the same
https://travis-ci.org/wikimedia/pywikibot-core/jobs/46549115
https://travis-ci.org/wikimedia/pywikibot-core/jobs/46581107
https://travis-ci.org/wikimedia/pywikibot-core/jobs/46652170
https://travis-ci.org/wikimedia/pywikibot-core/jobs/46673227

After installing a logger "socketIO_client" on , we see the error is:

WARNING:socketIO_client:[connection error] connection closed ()

Change 184285 had a related patch set uploaded (by John Vandenberg):
Prevent hang in LiveRCPageGenerator

https://gerrit.wikimedia.org/r/184285

Patch-For-Review

jayvdb mentioned this in T86506: rcstream doesnt log important status changes within a socketio connection.Jan 12 2015, 2:04 AM

Change 184285 merged by jenkins-bot:
Prevent hang in LiveRCPageGenerator

https://gerrit.wikimedia.org/r/184285

There seem to be two issues here:

SocketIO_Client's handling of heartbeats (as discussed a few days ago on IRC). https://github.com/invisibleroads/socketIO-client/blob/master/socketIO_client/__init__.py#L298 handles the case where heartbeat_interval >> .wait() interval, but not the other way around.
- One possible way to solve this is by adding an on_hearbeat handler, replying to the servers' heartbeat instead of sending our own.
on reconnect, on_connect is not called. This could be related to T86771: RCStream should remember subscriptions after socket.io re-connects, but in our case, the pypi version is used, which means https://github.com/invisibleroads/socketIO-client/commit/ee425459bd7d98f2df45b63795852420e7950720 is not actually part of the code base yet :/

Change 185200 had a related patch set uploaded (by Merlijn van Deen):
RCStream: return heartbeats and handle on_reconnect

https://gerrit.wikimedia.org/r/185200

Patch-For-Review

the reason on_connect is not called is the following. Compare

DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling
DEBUG:socketIO_client.transports:[transport selected] websocket
DEBUG:socketIO_client.transports:[packet sent] 1::/rc:
DEBUG:socketIO_client.transports:[packet received] 1::
DEBUG:socketIO_client: [connect]
DEBUG:socketIO_client.transports:[packet received] 1::/rc

DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling
DEBUG:socketIO_client.transports:[packet received] 1::
DEBUG:socketIO_client: [connect]
(... nothing)

This means socketio_client doesn't actually register in the /rc namespace, and thus our code is not called.

Change 185200 merged by jenkins-bot:
RCStream: return heartbeats and handle on_reconnect

https://gerrit.wikimedia.org/r/185200

This appears to be fixed.

Okay since this test (which was 7 days ago) all Python 3.3 tests which used setup.py stalled (the nosetests ran fine). It seems that all tests errored the same way. Maybe this is unrelated to this bug, but I'm not familiar with socketIO so I can't really determine that. A new one should be opened then.

Interestingly since then Travis is installing requests-2.5.3 and socketIO-client is using that library so they could be connected.

XZise mentioned this in T91394: requests 2.5.3 is causing issues on import.Mar 3 2015, 2:25 PM

Jdforrester-WMF removed a project: Patch-For-Review.Mar 3 2015, 5:27 PM

jayvdb mentioned this in rPWBCe39165a1a99b: Prevent hang in LiveRCPageGenerator.Mar 4 2015, 8:46 AM

jayvdb mentioned this in rPWBCba4e1e2ef10e: RCStream: return heartbeats and handle on_reconnect.

Regarding my comment 9 days ago: That is fixed and was caused because of T91393#1104004.

LiveRCPageGenerator using socketIO_client stall the builds regularlyClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

LiveRCPageGenerator using socketIO_client stall the builds regularly
Closed, ResolvedPublic
Actions