Page MenuHomePhabricator
Create Task
Maniphest T68232

Upgrade RCStream backend to use socket.io 1.0 protocol
Closed, DeclinedPublic
Actions

Description

When using socket.io.js from 1.0.x, no connection can be established.

The client is trying to make connections to

http://stream.wikimedia.org/socket.io/?EIO=2&transport=polling&t=1402016180378-0

Which are all 404. So I guess this is part of the new protocol that is not compatible. I don't know whether 1.0.x intends to be incompatible with the old protocol, or whether it is failing the to the CORS error (which makes it impossible for it to see that it is a 404).

Version: wmf-deployment
Severity: enhancement
URL: https://github.com/abourget/gevent-socketio/issues/192

Details

Reference
bz66232

Related Objects
Search...

StatusSubtypeAssignedTask
 DeclinedXqtT125197 Give hint to the current socketIO_client in ImportError of rcstream.py
 DeclinedNoneT91393 RCStream is not accessible from python client due to using socket-io 1.0 while only socket-io 0.9 is offered
 DeclinedNoneT68232 Upgrade RCStream backend to use socket.io 1.0 protocol
 ResolvedKrinkleT86803 Rewrite RCStream in Node.js
 DeclinedKrinkleT121722 Deploy node-rcstream
 ResolvedOttomataT130651 EventStreams
 Duplicate• PcheloloT137379 Replace kafka-node with more mature bindings, ideally using librdkafka
 Resolved• PcheloloT140284 Develop a node kafka driver based on librdkafka
 ResolvedOttomataT145095 Kasocki Prototype
 Resolved• NuriaT145099 Write node-rdkafka event.stats callback that reports stats to statsd
 ResolvedOttomataT145140 Run Kasocki tests in Jenkins via Differential commits
 ResolvedOttomataT148470 Kafka SSE Prototype
 ResolvedOttomataT148651 Create KafkaSSE diffusion repository
 ResolvedOttomataT148779 Prepare eventstreams (with KafkaSSE) for deployment
 ResolvedOttomataT143925 Productionize and deploy Public EventStreams
 DeclinedOttomataT145805 Support node cluster sticky-session in service-runner
 DeclinedOttomataT148043 Fix consumer.disconnect() node-rdkafka bug
 ResolvedOttomataT149736 Bikeshed what events should be exposed in public EventStreams API
 ResolvedOttomataT167670 Expose revision-create in EventStreams
 ResolvedOttomataT167180 Emit revision-score event to EventBus and expose in EventStreams
 ResolvedHalfakT172566 Include label-specific schemas with model_info
 ResolvedJAllemandouT150436 Improve tests for KafkaSSE
 DeclinedNoneT150439 Tests for swagger spec stream routes in EventStreams
 ResolvedOttomataT152030 RecentChanges in Kafka
 OpenNoneT152731 Implement server side filtering for EventStreams (if we should)
 ResolvedOttomataT153117 EventStreams documentation
 ResolvedmfornsT154328 Set charset=utf-8 in Content-Type response header from sse.js client
 ResolvedOttomataT156919 Port RCStream clients to EventStreams
 ResolvedXqtT158943 Implement EventStreams client for pywikibot
 DuplicateNoneT160997 Migrate pywikibot from RCStream to EventStreams
 Resolved• CmjohnsonT170157 decommission rcs100[12]
 ResolvedXqtT170534 Decommission RCStream in pywikibot framework
 ResolvedOttomataT157492 Report number of stream connections to statsd
 ResolvedOttomataT158106 Update EventBus RCFeed config to use newly refactored settings
 ResolvedOttomataT158066 Create EventStreams swagger spec docs endpoint
 ResolvedOttomataT159179 Create /schema/:schema endpoint in eventbus service to serve schemas by schema_uri
 ResolvedOttomataT160030 old revision length missing in /v2/stream/recentchange
 ResolvedOttomataT160080 EventStreams Blog Post
 DeclinedOttomataT160748 Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus
 DeclinedOttomataT161027 Puppetize event schema topic configuration
 ResolvedOttomataT172356 Decom RCStream in Beta Cluster

Event Timeline

• bzimport raised the priority of this task from to High.Nov 22 2014, 3:19 AM
• bzimport added a project: EventStreams.
• bzimport set Reference to bz66232.
Krinkle created this task.Jun 6 2014, 12:58 AM
ori added a comment.Jun 6 2014, 2:57 AM

v1.0 changes the protocol substantially. Apart from bragging rights about using the latest and greatest, I don't see major gains to be had. 0.9 is well-tested. IMO, we should upgrade in lock-step with what is available in Debian Testing.

ori added a comment.Jun 6 2014, 2:58 AM

(Not to mention: we want to give the popular client libraries time to upgrade, too.)

Krenair subscribed.Jan 11 2015, 3:25 AM

This sounds very similar to T78463...

Krenair mentioned this in T69955: Stream: Websocket client fails on ws://stream.wikimedia.org/rc: HTTP/1.1 404 Not Found.Jan 11 2015, 3:25 AM
Krinkle added a comment.Jan 14 2015, 7:17 PM
In T78463#962429, @Krinkle wrote:

Lowering priority since there's no expectation that RCStream will work with any protocol version. socket.io has multiple protocol versions and generally a server should only support one.

The v0.9 protocol is by far the most popular at the moment, though that is slowly changing now that at least the reference implementation (socket.io-client) has a v1.0 release. However you can't use that yet. And installing the "latest" version of something isn't a good practice anyway. The correct version is now documented https://wikitech.wikimedia.org/wiki/RCStream#JavaScript. The majority of changes between v0.9 and v1.0 is in the backend. The API surface is largely the same and shouldn't block any particular features.

I do however very much want RCStream to be ahead of the curve and encouraging new software to be written in current versions of socket.io and not old ones. RCStream is currently implemented in Python and uses the gevent-socketio package. That package has not yet implemented support for the socket.io v1.0 protocol, so there's no upgrade available. It's been raised upstream (https://github.com/abourget/gevent-socketio/issues/192) however it seems the project is no longer seeing active maintenance.

Considering RCStream is a fairly small library, I'd recommend taking this opportunity to rewrite it in Node.js using the official socket.io server implementation (which is written in node) in favour of using a port.

Krinkle renamed this task from RCStream: Upgrade backend to use protocol that works with socket.io 1.0.x to Upgrade RCStream backend to use socket.io 1.0 protocol.Jan 14 2015, 7:18 PM
Krinkle lowered the priority of this task from High to Medium.
Krinkle merged a task: T78463: RCStream: Upgrade server protocol to socket.io v1.
Krinkle set Security to None.
Krinkle added subscribers: Mitar, Aklapper.
Krinkle added a subtask: T86803: Rewrite RCStream in Node.js.Jan 14 2015, 7:21 PM
Krinkle mentioned this in T91393: RCStream is not accessible from python client due to using socket-io 1.0 while only socket-io 0.9 is offered.Mar 3 2015, 3:49 PM
Ricordisamoa subscribed.Apr 10 2015, 10:21 PM
Krinkle moved this task from Backlog to Next up on the EventStreams board.Jul 23 2015, 7:31 PM
Krinkle claimed this task.Jul 23 2015, 8:25 PM
Krinkle moved this task from Next up to Assigned on the EventStreams board.
Krinkle added a parent task: T91393: RCStream is not accessible from python client due to using socket-io 1.0 while only socket-io 0.9 is offered.
Sitic subscribed.Jul 29 2015, 2:39 AM
Ebrahim subscribed.Aug 14 2015, 11:07 AM
jayvdb subscribed.Oct 15 2015, 11:51 AM
jayvdb added a comment.Oct 15 2015, 12:06 PM

Note there is a Python rewrite which provides socket.io 1.0 : https://github.com/shuoli84/gevent_socketio2 . However: no commits since November 28, 2014

A better Python option : https://github.com/miguelgrinberg/python-socketio

Afandian subscribed.Oct 26 2015, 5:34 PM
Glaisher subscribed.Dec 15 2015, 4:09 PM
Daniel_Mietchen subscribed.Dec 16 2015, 2:11 AM
Krinkle closed subtask T86803: Rewrite RCStream in Node.js as Resolved.Dec 16 2015, 11:26 PM
Krinkle added a subtask: T121722: Deploy node-rcstream.
Krinkle changed the task status from Open to Stalled.Dec 17 2015, 12:23 AM
Krinkle removed Krinkle as the assignee of this task.
Tarrow subscribed.Dec 21 2015, 1:52 PM
Xqt added a parent task: T125197: Give hint to the current socketIO_client in ImportError of rcstream.py.Jan 29 2016, 9:42 AM
Krinkle mentioned this in T87780: Add notice about rcstream to IRC RC motd.Feb 16 2016, 2:46 PM
Krinkle mentioned this in T127181: [CatWatch] Make it easy to distinguish CatWatch changes on IRC.Feb 23 2016, 2:48 PM
Krinkle closed subtask T121722: Deploy node-rcstream as Declined.Apr 21 2016, 11:27 PM
Krinkle added a subtask: T130651: EventStreams.Apr 21 2016, 11:30 PM
edwardbetts subscribed.May 22 2016, 8:28 AM
utdrmac subscribed.Dec 7 2016, 6:20 PM

Can this be re-evaluated and prioritized? socket.io has been 1.0+ for over a year and a half; it's no longer "new". All official client libraries supplied by socket.io only support 1.0+ now.

• OldUser02 awarded a token.Dec 17 2016, 1:57 PM
• OldUser02 subscribed.
Fluff subscribed.Jan 29 2017, 12:57 AM
Krinkle added a comment.Jan 30 2017, 10:52 PM

I'm closing this task and T91393 since it's unlikely this will happen. Rather than upgrading RCSteam to Socket.IO v1.0 (which would be a breaking change for existing RCStream consumers), I'll instead help push forward T130651: EventStreams.

Krinkle closed this task as Declined.Jan 30 2017, 10:52 PM
Xqt mentioned this in T125197: Give hint to the current socketIO_client in ImportError of rcstream.py.Feb 24 2017, 5:39 AM
Ottomata closed subtask T130651: EventStreams as Resolved.Mar 1 2018, 6:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits