Page MenuHomePhabricator

Make node-rdkafka an optional dependency of EventGate
Open, HighPublic

Description

Producing to Kafka is just one way to use EventGate. It's nice to have it as the default, except that building the librdkafka binary is not very lightway, nor does it always work so smoothly on all OS-es (e.g. MacOS). By making node-rdkafka an optional dependency (and guarding against it being missing), we should be able to more easily create development eventgate instances without needing to build librdkafka.

Event Timeline

Ottomata created this task.Oct 20 2020, 6:37 PM

Change 635384 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Bump eventgate sha to 3b6023c to get node-rdkafka as an optional dependency

https://gerrit.wikimedia.org/r/635384

fdans triaged this task as High priority.Mon, Oct 26, 3:58 PM
fdans moved this task from Incoming to Event Platform on the Analytics board.

Change 637026 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[node-rdkafka-factory@master] Initial commit

https://gerrit.wikimedia.org/r/637026

Ottomata added a subscriber: Milimetric.EditedWed, Oct 28, 9:00 PM

Ok, I did some work today to factory out our async Kafka factory wrapper from eventgate into its own library. This makes it easier to make it an optional dependency of both eventgate and eventgate-wikimedia. This currently includes 4 changes, including one to make it all work in EventLogging as a devserver, for the parent task T259202.

These all need review, gonna bug @Milimetric and/or @Pchelolo for the first 3, and @Mholloway for the last EventLogging one.

  1. New node-rdkafka-factory repo and module: https://gerrit.wikimedia.org/r/c/node-rdkafka-factory/+/637026
  2. Make eventgate use node-rdkafka-factory: https://github.com/wikimedia/eventgate/pull/12
  3. Make eventgate-wikimedia use node-rdkafka-factory: https://gerrit.wikimedia.org/r/c/eventgate-wikimedia/+/635384
  4. Make EventLogging use eventgate-wikimedia without node-rdkafka: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/617770

These all need to be merged and published in order.

Change 637514 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[integration/config@master] Add CI for node-rdkafka-factory

https://gerrit.wikimedia.org/r/637514

Change 637514 merged by jenkins-bot:
[integration/config@master] Add CI for node-rdkafka-factory

https://gerrit.wikimedia.org/r/637514

Mentioned in SAL (#wikimedia-releng) [2020-10-29T16:57:39Z] <James_F> Zuul: Add CI for node-rdkafka-factory T266058

Change 637026 merged by jenkins-bot:
[node-rdkafka-factory@master] Initial commit

https://gerrit.wikimedia.org/r/637026

Change 635384 merged by Ottomata:
[eventgate-wikimedia@master] Bump eventgate to 1.5.0 to get node-rdkafka-factory as optional dependency

https://gerrit.wikimedia.org/r/635384

Change 637748 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Fix require of namespaced @Wikimedia/node-rdkafka-factory

https://gerrit.wikimedia.org/r/637748

Change 637748 merged by Ottomata:
[eventgate-wikimedia@master] Fix require of namespaced @Wikimedia/node-rdkafka-factory

https://gerrit.wikimedia.org/r/637748

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Mon, Nov 16, 8:59 PM