Since we rewrote the EventLogging kafka reader handler to use pykafka (for the BalancedConsumer), move the writer to the same library for consistency.
- do a quick performance analysis to make sure there is no huge difference
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| [WIP] Change kafka writer to use pykafka Producer | mediawiki/extensions/EventLogging | master | +12 -60 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • kevinator | T102225 {stag} EventLogging on Kafka | |||
| Declined | • madhuvishy | T109244 Move Eventlogging Kafka writer to use pykafka's Producer instead of python-kafka {stag} [8 pts] |
Event Timeline
Comment Actions
Change 232408 had a related patch set uploaded (by Madhuvishy):
[WIP] Change kafka writer to use pykafka Producer
Comment Actions
Looked into this a bit more with @Ottomata, and it looks like atm, python-kafka's producer is more robust that pykafka's for our purpose.
Reasons:
- Pykafka forces us to create instances of a topic and a producer for every message, these are not cached by the library, and it's hard to cache this on our end because it ties producers to a topic unlike python-kafka.
- Pykafka has a sync producer unlike python-kafka, which slows things down a bit - they have added an Async one in the last few days - but since it's fairly new, we don't want to jump to using it.
Comment Actions
Change 232408 abandoned by Madhuvishy:
[WIP] Change kafka writer to use pykafka Producer
Reason:
We decided not to move the writer to pykafka