Page MenuHomePhabricator

Deploy node-rcstream
Closed, DeclinedPublic

Description

  • Make sure node-rcstream is up to date with any API changes in python-rcstream since last year.
  • Convert to use node-servicerunner.
  • Puppetize the implementation and test it out in labs.
    • Initially behind a /v2/ to test it out, later to replace python-rcstream.

Event Timeline

Krinkle created this task.Dec 16 2015, 11:08 PM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added a project: Wikimedia-Stream.
Krinkle added subscribers: Krinkle, ori, marcoil.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 16 2015, 11:08 PM
Krinkle claimed this task.Dec 16 2015, 11:25 PM
Krinkle triaged this task as High priority.
Krinkle set Security to None.
Krinkle moved this task from Inbox to Assigned on the Wikimedia-Stream board.
jayvdb added a subscriber: jayvdb.Dec 17 2015, 3:39 AM

Change 271805 had a related patch set uploaded (by Krinkle):
Match latest python-rcstream implementation

https://gerrit.wikimedia.org/r/271805

Change 271805 merged by jenkins-bot:
Match latest python-rcstream implementation

https://gerrit.wikimedia.org/r/271805

Krinkle closed this task as Declined.EditedApr 21 2016, 11:27 PM

The current node-rcstream implementation is up to date in theory, but has never been tested with an actual RedisRCFeed backend. Not locally, not in beta, not in prod.

Even if it works, it's not up to production standards (e.g. doesn't use service-runner). So it'd be relatively hard to scale, hard to monitor/debug, and overall uptime automation suboptimal.

I'm closing this task since it doesn't seem worth investing further in. Rather than working on this (which will be a breaking change for existing RCStream consumers), I'll instead help push forward T130651: EventStreams.

That project will supersede RCStream and result in the creation of a Socket.IO service very similar to RCStream. That service will be written in Node.js and use the stable Socket.IO v1.0 protocol.

I'll recommend that service to be hosted on the stream.wikimedia.org domain.

The public event stream T130651 will effectively become RCStream 2.0 (though under a new name for the software library, as RCStream doesn't cover the scope).

The current node-rcstream implementation is up to date in theory, but has never been tested with an actual RedisRCFeed backend. Not locally, not in beta, not in prod.

Wait what? To clarify, we're going to abandon rcstream, just because no one has ever tested it locally? Obviously it hasn't happened in beta or prod yet, that's the point of this ticket.

Even if it works, it's not up to production standards (e.g. doesn't use service-runner). So it'd be relatively hard to scale, hard to monitor/debug, and overall uptime automation suboptimal.

If adapting it to use the service-runner framework needs to happen, that should have been filed as a blocker or something.

I'm closing this task since it doesn't seem worth investing further in. Rather than working on this (which will be a breaking change for existing RCStream consumers), I'll instead help push forward T130651: EventStreams.
That project will supersede RCStream and result in the creation of a Socket.IO service very similar to RCStream. That service will be written in Node.js and use the stable Socket.IO v1.0 protocol.

"very similar to RCStream" ...then why not just modify RCStream? Given how close we are to having a proper machine readable rcfeed, I'd be pretty disappointed if we just gave up on it and decided to start over from scratch again.

Krinkle added a comment.EditedApr 22 2016, 12:19 AM

To clarify, we're going to abandon rcstream, just because no one has ever tested it locally?
[..]
"very similar to RCStream" ...then why not just modify RCStream?

To clarify, I'm talking here about abandoning node-rcsteam. Which at this point is nothing but a personal Git repo with a 200-line JS file I wrote once in 2014 based on current production RCStream (written in Python). It was blindly written and never tested or used anywhere.

Apart from basic code you can find in any Node.js and Socket.IO tutorial, there is nothing to salvage there.

We'll probably re-use some logic from the current python rcstream (mediawiki/services/rcstream) but that's pretty basic too and mostly just boilerplate for redis and gevent-socketio. Neither of which are relevant for the planned Kafka-backed Node.js Socket.IO service.

So instead of wasting a lot of effort into essentially continue development of node-rcstream (which has only just gotten started, there is nothing to modify, I wrote it in a day in 2014 and has never left Git), instead I'm gonna put that effort in the similar Node.js service Analytics is building.

Even if we were to continue the idea of upgrading the current rcstream to Socket.IO v1.0, it'd be a breaking change for existing clients. I'd rather not have another breaking change when the public event stream is ready.

To rephrase my original comment:

  • Right now we have RedisRCFeed in production with python-rcstream providing a Socket.IO v0.9 protocol service.
  • We're gonna have recentchanges in Kafka pretty soon (probably via a HttpRCFeed submitting to the EventBus proxy to Kafka).
  • RCStream is gonna be the only consumer of RedisRCFeed and would make sense to port to consume from Kafka.
  • RCStream is unstable and hard to use due to the deprecated 0.9 protocol. The plan was to port it to Node.js and Socket.IO.
  • Analytics wants a Socket.IO service that exposes various events, not just recentchanges.
  • This service would have an almost identical protocol, API and feature-set as the rewritten node-rcstream-on-kafka.

Conclusion: Let's build that Node.js service together.

That project will supersede RCStream and result in the creation of a Socket.IO service very similar to RCStream. That service will be written in Node.js and use the stable Socket.IO v1.0 protocol.

Aren't you doing enemy-of-the-good here?
How much of donors' money has been wasted.