Page MenuHomePhabricator

Prepare eventstreams (with KafkaSSE) for deployment
Closed, ResolvedPublic21 Estimated Story Points

Description

  • create mediawiki/services/eventstreams repo
  • create mediawiki/services/eventstreams/deploy repo with scap config and node_modules (figure out how services builds with librdkafka .deb)
  • puppetize eventstreams
  • test in deployment-prep
  • figure out where this will be deployed (in REST API hierarchy? with varnish? restbase?)

Event Timeline

Ottomata created this task.Oct 20 2016, 7:01 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
  • create mediawiki/services/eventstreams/deploy repo with scap config and node_modules (figure out how services builds with librdkafka .deb)

We use service-runner to build the deploy repo, and pull in librdkafka from WMF's APT repo by [specifying it in package.json](https://github.com/wikimedia/change-propagation/blob/1ff4e2ae9cb0be331b2f8b2adc83eb58e3c5d0e4/package.json#L70-L83).

  • puppetize eventstreams

That should be as easy as defining the correct service::node resource. We can help with that.

  • figure out where this will be deployed (in REST API hierarchy? with varnish? restbase?)

Definitely in the REST hierarchy. Given its streaming architecture, we need to weigh carefully where exactly to put it, but my first instinct would be to bypass Varnish altogether if possible not to clog connections.

  • create mediawiki/services/eventstreams/deploy repo with scap config and node_modules (figure out how services builds with librdkafka .deb)

We use service-runner to build the deploy repo, and pull in librdkafka from WMF's APT repo by [specifying it in package.json](https://github.com/wikimedia/change-propagation/blob/1ff4e2ae9cb0be331b2f8b2adc83eb58e3c5d0e4/package.json#L70-L83).

Also a very important line: https://github.com/wikimedia/change-propagation/blob/1ff4e2ae9cb0be331b2f8b2adc83eb58e3c5d0e4/package.json#L68

Change 317874 had a related patch set uploaded (by Ottomata):
Eventstreams with KafkaSSE initial commit on service-template

https://gerrit.wikimedia.org/r/317874

Change 317883 had a related patch set uploaded (by Ottomata):
deploy repo setup

https://gerrit.wikimedia.org/r/317883

Change 317981 had a related patch set uploaded (by Ottomata):
[WIP] evenstreams puppetization

https://gerrit.wikimedia.org/r/317981

Milimetric set the point value for this task to 21.Oct 27 2016, 3:51 PM

Change 317874 merged by Ottomata:
Eventstreams with KafkaSSE initial commit on service-template

https://gerrit.wikimedia.org/r/317874

Change 317883 merged by Ottomata:
deploy repo setup

https://gerrit.wikimedia.org/r/317883

Change 319129 had a related patch set uploaded (by Ottomata):
Use Ruby yaml lib to render node::service deployment vars

https://gerrit.wikimedia.org/r/319129

Change 319129 had a related patch set uploaded (by Ottomata):
Use ordered_yaml function to render node::service deployment vars

https://gerrit.wikimedia.org/r/319129

Change 319129 merged by Ottomata:
Use ordered_yaml function to render node::service deployment vars

https://gerrit.wikimedia.org/r/319129

Change 317981 merged by Ottomata:
eventstreams puppetization

https://gerrit.wikimedia.org/r/317981

Yes! Deployed to beta via puppet and scap.

This works!
http://codepen.io/ottomata/pen/VKNyEw?editors=0011

Now waiting on node-rdkafka to make a release.

Nuria closed this task as Resolved.Nov 29 2016, 7:53 PM