RCStream should remember subscriptions after socket.io re-connects
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	Petrb
	Jan 14 2015, 3:57 PM

Description

I am using that new RC feed in some python script, basically just a copy of https://wikitech.wikimedia.org/wiki/RCStream

I randomly get disconnected with this error:

WARNING:socketIO_client:[connection error] connection closed ()
DEBUG:socketIO_client: [disconnect]
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
WARNING:socketIO_client:[waiting for connection] unexpected status code (502)
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 200 89
DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling
DEBUG:socketIO_client.transports:[transport selected] websocket
DEBUG:socketIO_client.transports:[packet received] 1::
DEBUG:socketIO_client: [connect]
DEBUG:socketIO_client.transports:[packet sent] 2:::
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client.transports:[packet sent] 2:::
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client.transports:[packet sent] 2:::

Only solution is restart of script. Why is that and how to fix it?

Related Objects

Mentioned In: T85717: LiveRCPageGenerator using socketIO_client stall the builds regularly
Mentioned Here: T156919: Port RCStream clients to EventStreams

Event Timeline

Petrb created this task.Jan 14 2015, 3:57 PM

Petrb raised the priority of this task from to Needs Triage.

Petrb updated the task description. (Show Details)

Petrb added a project: MediaWiki-Recent-changes.

Petrb subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 14 2015, 3:57 PM

yuvipanda added subscribers: Krinkle, ori, yuvipanda.Jan 14 2015, 3:59 PM

Petrb set Security to None.Jan 14 2015, 4:00 PM

Petrb added a subscriber: hashar.

out of curiosity here is the python script https://github.com/huggle/XMLRCS/blob/master/src/ws2r/ws2r.py

The reconnection looks succesful, but you're not automatically re-subscribed to the relevant feeds. I *think* socket.io is supposed to take care of this, but I'm not completely sure. @ori, could that be related to the load balancing again?

ok, how do I automatically do that

is there any documentation for that python library?

Krenair edited projects, added EventStreams; removed MediaWiki-Recent-changes.Jan 14 2015, 5:23 PM

https://github.com/invisibleroads/socketIO-client

You can probably add an on_reconnect handler. Subscribe is idempotent, so should be safe even if you're not actually supposed to re-subscribe.

Aklapper renamed this task from New RC feed is randomly failing to New RC feed is randomly failing: socket.io not re-subscribed?.Jan 14 2015, 6:05 PM

Aklapper triaged this task as Medium priority.

Now that I take a better look: socketIO-client is doing a full new handshake: instead of reconnecting with the same session, it requests a new session:

DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 200 89

so the server has no way to know you're the same client as before.

As such, you'll need to re-subscribe to the stream.

In T86771#976617, @valhallasw wrote:

Now that I take a better look: socketIO-client is doing a full new handshake: instead of reconnecting with the same session, it requests a new session:

fwiw, http://codepen.io/Krinkle/full/laucI/ also does this, and that uses the official (js) client. I think the issue is rather with SocketIO_client not calling on_connect, and as far as I can see, this is only very recent.

hashar unsubscribed.Jan 14 2015, 8:47 PM

Ok I did that, it failed again though during night, but I have no logs, so let's see now

Here we go

WARNING:socketIO_client:[connection error] [Errno 104] Connection reset by peer
DEBUG:socketIO_client: [disconnect]
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 200 89
DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling
DEBUG:socketIO_client.transports:[transport selected] websocket
DEBUG:socketIO_client.transports:[packet received] 1::
DEBUG:socketIO_client: [connect]
DEBUG:socketIO_client.transports:[packet sent] 2:::
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client: [disconnect]
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 200 89
DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling
DEBUG:socketIO_client.transports:[transport selected] websocket
DEBUG:socketIO_client.transports:[packet received] 1::
DEBUG:socketIO_client: [connect]
DEBUG:socketIO_client.transports:[packet sent] 2:::
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
WARNING:socketIO_client:[connection error] connection closed ()
DEBUG:socketIO_client: [disconnect]
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 200 89
DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling
DEBUG:socketIO_client.transports:[transport selected] websocket
DEBUG:socketIO_client.transports:[packet received] 1::
DEBUG:socketIO_client: [connect]
DEBUG:socketIO_client.transports:[packet sent] 2:::
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client: [disconnect]
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 200 89
DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling
DEBUG:socketIO_client.transports:[transport selected] websocket
DEBUG:socketIO_client.transports:[packet received] 1::
DEBUG:socketIO_client: [connect]
DEBUG:socketIO_client.transports:[packet sent] 2:::
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]

New source has:

def on_reconnect(self, *args):
  self.emit('subscribe', '*')
def on_connect(self):
  self.emit('subscribe', '*')

it doesn't seem to work :(

yuvipanda unsubscribed.Jan 15 2015, 4:43 PM

valhallasw mentioned this in T85717: LiveRCPageGenerator using socketIO_client stall the builds regularly.Jan 15 2015, 4:50 PM

For those of us patrolling for SWMT and on spam patrol the use of the bot and reliability of feeds is important. Support for resolution of the matter would be appreciated/

btw I added some debug prints to both on_connect and on_reconnect, but neither triggered on another disconnect I had now:

WARNING:socketIO_client:[connection error] connection closed ()
DEBUG:socketIO_client: [disconnect]
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
WARNING:socketIO_client:[waiting for connection] unexpected status code (502)
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 502 181
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wikimedia.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /socket.io/1/ HTTP/1.1" 200 89
DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling
DEBUG:socketIO_client.transports:[transport selected] websocket
DEBUG:socketIO_client.transports:[packet received] 1::
DEBUG:socketIO_client: [connect]
DEBUG:socketIO_client.transports:[packet sent] 2:::
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]
DEBUG:socketIO_client.transports:[packet sent] 2:::
DEBUG:socketIO_client.transports:[packet received] 2::
DEBUG:socketIO_client: [heartbeat]

That means on_reconnect nor on_connect was called after reconnect, why?

Upstream: https://github.com/invisibleroads/socketIO-client/issues/61

I'm having similar issue of not getting updates after ~1-2 hours of running with live RC pagegenerator.
From log:

WARNING: socketIO_client:[connection error] connection closed () WARNING:socketIO_client:[waiting for connection] unexpected status code (502)

In T86771#981361, @Billinghurst wrote:

For those of us patrolling for SWMT and on spam patrol the use of the bot and reliability of feeds is important. Support for resolution of the matter would be appreciated/

Which SWMT patrol tool is using this at the moment? Handling of subscription is a very basic thing. I can't imagine a tool being deployed out there currently that is used by users and does not handle this correctly.

@Billinghurst: Any idea about Krinkle's question?

My comment was about the utilisation of wmbot in various places, and it is about direct and secondary watches rather than the other bots that work in the CVN space.

Yes, wm-bot uses RCStream right now and it is used by many users. It uses it over XmlRcs which fortunately can now handle this problem using a nasty workaround.

I managed to write a python script in a way that when it gets disconnected it crashes, and gets restarted, which is only solution that works so far and even if it's far from being perfect as it causes small (1 - 6 sec) outages in stream every 2 - 10 hours, it can at least provide stream of changes without failure.

On other hand even if I was able to handle disconnects gracefully this still isn't good because in first place these failures shouldn't happen so often.

What is causing these disconnects? Is it bad connectivity between labs and production? I doubt so because it happens to me even when I am using RCStream python or js script on my local PC. This problem is happening on both JS and python versions of client.

BTW connection to XmlRcs seems to have no issues so I doubt there is actually any problem with connectivity and it's being constantly used by approximately 10 - 20 users who are connected to it

Krinkle moved this task from Backlog to Backlog on the EventStreams board.Jul 23 2015, 8:28 PM

Krinkle renamed this task from New RC feed is randomly failing: socket.io not re-subscribed? to RCStream should remember subscriptions after socket.io re-connects.Dec 16 2015, 11:27 PM

Krinkle moved this task from Backlog to Backlog-RCStream on the EventStreams board.Jan 30 2017, 10:59 PM

Declining per RCStream being deprecated and scheduled to be shut down on July 7th.

RCStream should remember subscriptions after socket.io re-connectsClosed, DeclinedPublicActions

Description

Related Objects

Event Timeline

RCStream should remember subscriptions after socket.io re-connects
Closed, DeclinedPublic
Actions