- Make sure node-rcstream is up to date with any API changes in python-rcstream since last year.
- Convert to use node-servicerunner.
- Puppetize the implementation and test it out in labs.
- Initially behind a /v2/ to test it out, later to replace python-rcstream.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Match latest python-rcstream implementation | rcstream | master | +240 -195 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | Xqt | T125197 Give hint to the current socketIO_client in ImportError of rcstream.py | |||
| Declined | None | T91393 RCStream is not accessible from python client due to using socket-io 1.0 while only socket-io 0.9 is offered | |||
| Declined | None | T68232 Upgrade RCStream backend to use socket.io 1.0 protocol | |||
| Declined | Krinkle | T121722 Deploy node-rcstream |
Event Timeline
Comment Actions
Change 271805 had a related patch set uploaded (by Krinkle):
Match latest python-rcstream implementation
Comment Actions
The current node-rcstream implementation is up to date in theory, but has never been tested with an actual RedisRCFeed backend. Not locally, not in beta, not in prod.
Even if it works, it's not up to production standards (e.g. doesn't use service-runner). So it'd be relatively hard to scale, hard to monitor/debug, and overall uptime automation suboptimal.
I'm closing this task since it doesn't seem worth investing further in. Rather than working on this (which will be a breaking change for existing RCStream consumers), I'll instead help push forward T130651: EventStreams.
That project will supersede RCStream and result in the creation of a Socket.IO service very similar to RCStream. That service will be written in Node.js and use the stable Socket.IO v1.0 protocol.
I'll recommend that service to be hosted on the stream.wikimedia.org domain.
The public event stream T130651 will effectively become RCStream 2.0 (though under a new name for the software library, as RCStream doesn't cover the scope).
Comment Actions
Wait what? To clarify, we're going to abandon rcstream, just because no one has ever tested it locally? Obviously it hasn't happened in beta or prod yet, that's the point of this ticket.
Even if it works, it's not up to production standards (e.g. doesn't use service-runner). So it'd be relatively hard to scale, hard to monitor/debug, and overall uptime automation suboptimal.
If adapting it to use the service-runner framework needs to happen, that should have been filed as a blocker or something.
I'm closing this task since it doesn't seem worth investing further in. Rather than working on this (which will be a breaking change for existing RCStream consumers), I'll instead help push forward T130651: EventStreams.
That project will supersede RCStream and result in the creation of a Socket.IO service very similar to RCStream. That service will be written in Node.js and use the stable Socket.IO v1.0 protocol.
"very similar to RCStream" ...then why not just modify RCStream? Given how close we are to having a proper machine readable rcfeed, I'd be pretty disappointed if we just gave up on it and decided to start over from scratch again.
Comment Actions
To clarify, I'm talking here about abandoning node-rcsteam. Which at this point is nothing but a personal Git repo with a 200-line JS file I wrote once in 2014 based on current production RCStream (written in Python). It was blindly written and never tested or used anywhere.
Apart from basic code you can find in any Node.js and Socket.IO tutorial, there is nothing to salvage there.
We'll probably re-use some logic from the current python rcstream (mediawiki/services/rcstream) but that's pretty basic too and mostly just boilerplate for redis and gevent-socketio. Neither of which are relevant for the planned Kafka-backed Node.js Socket.IO service.
So instead of wasting a lot of effort into essentially continue development of node-rcstream (which has only just gotten started, there is nothing to modify, I wrote it in a day in 2014 and has never left Git), instead I'm gonna put that effort in the similar Node.js service Analytics is building.
Even if we were to continue the idea of upgrading the current rcstream to Socket.IO v1.0, it'd be a breaking change for existing clients. I'd rather not have another breaking change when the public event stream is ready.
To rephrase my original comment:
- Right now we have RedisRCFeed in production with python-rcstream providing a Socket.IO v0.9 protocol service.
- We're gonna have recentchanges in Kafka pretty soon (probably via a HttpRCFeed submitting to the EventBus proxy to Kafka).
- RCStream is gonna be the only consumer of RedisRCFeed and would make sense to port to consume from Kafka.
- RCStream is unstable and hard to use due to the deprecated 0.9 protocol. The plan was to port it to Node.js and Socket.IO.
- Analytics wants a Socket.IO service that exposes various events, not just recentchanges.
- This service would have an almost identical protocol, API and feature-set as the rewritten node-rcstream-on-kafka.
Conclusion: Let's build that Node.js service together.