Page MenuHomePhabricator

Rewrite RCStream in Node.js
Closed, ResolvedPublic

Description

The v0.9 protocol is by far the most popular at the moment, though that is slowly changing now that at least the reference implementation (socket.io-client) has a v1.0 release. However you can't use that yet. And installing the "latest" version of something isn't a good practice anyway. The correct version is now documented https://wikitech.wikimedia.org/wiki/RCStream#JavaScript. The majority of changes between v0.9 and v1.0 is in the backend. The API surface is largely the same and shouldn't block any particular features.
I do however very much want RCStream to be ahead of the curve and encouraging new software to be written in current versions of socket.io and not old ones. RCStream is currently implemented in Python and uses the gevent-socketio package. That package has not yet implemented support for the socket.io v1.0 protocol, so there's no upgrade available. It's been raised upstream (https://github.com/abourget/gevent-socketio/issues/192) however it seems the project is no longer seeing active maintenance.
Considering RCStream is a fairly small library, I'd recommend taking this opportunity to rewrite it in Node.js using the official socket.io server implementation (which is written in node) in favour of using a port.

Event Timeline

Krinkle created this task.Jan 14 2015, 7:20 PM
Krinkle claimed this task.
Krinkle raised the priority of this task from to High.
Krinkle updated the task description. (Show Details)
Krinkle added a project: Wikimedia-Stream.
Krinkle added subscribers: Krinkle, ori.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 14 2015, 7:20 PM

Apparently I did this last year:
https://github.com/Krinkle/node-rcstream

I had forgotten about it though. It looks quite similar implementation wise. I can't stay in what state it is, but it should make for a good starting point.

Krinkle removed Krinkle as the assignee of this task.Mar 4 2015, 5:49 PM
Krinkle lowered the priority of this task from High to Normal.
Krinkle set Security to None.
Krinkle moved this task from Inbox to Next up on the Wikimedia-Stream board.Jul 23 2015, 7:30 PM
jayvdb added a subscriber: jayvdb.Oct 15 2015, 11:51 AM
Afandian added a subscriber: Afandian.EditedOct 26 2015, 4:34 PM

This looks like a good opportunity to include diff information. A number of bots are watching the RCStream and immediately fetching relevant diff. This is expensive for everyone, and having a new RCStream include this feature as an option would be great. See T100082

Krinkle added a comment.EditedOct 27 2015, 1:27 AM

This looks like a good opportunity to include diff information. A number of bots are watching the RCStream and immediately fetching relevant diff. This is expensive for everyone, and having a new RCStream include this feature as an option would be great. See T100082

I appreciate the enthusiasm but this isn't an opportunity for that. The RCStream software is pure middle-ware to publish the internal RCFeed to the outside world using the Socket.IO protocol. It doesn't have access to diffs. It will require added complexity in MediaWiki and RCFeed to make this work via RCStream. RCFeed is not meant to compute or distribute such large blobs.

Let's focus our efforts on RESTBase instead. Its low latency and deterministic scheme will cancel out most reasons for why fetching diffs is unattractive right now. Using RESTBase also requires less engineering effort to implement diffs and provides a more stable, maintainable and extendable platform.

Thanks @Krinkle. It was worth asking... Looking forward to seeing a 1.0 version. Looks like library support in a few languages is good for 1.0.

Krinkle closed this task as Resolved.Dec 16 2015, 11:25 PM
Krinkle claimed this task.
Krinkle moved this task from Next up to Assigned on the Wikimedia-Stream board.