Page MenuHomePhabricator

Support kafka in eventlogging client on terbium
Closed, ResolvedPublic

Description

The eventlogging client python module installed on tin.eqiad.wmnet doesn't support Kafka.

Which more and more stuff moving away from ZMQ, this makes it impossible to test or debug event streams. Not unless there is another host I can access that does have a working eventlogging-kafka client.

It should probably also be installed on terbium instead of tin to discourage running of such processes on the deployment host.


Event Timeline

Krinkle raised the priority of this task from to High.
Krinkle updated the task description. (Show Details)
Krinkle subscribed.

I don't think we even knew that eventlogging was deployed to tin. Both terbium and tin are precise. Is there a Trusty host where we can install this?

We should install it on stat1002.

Change 238963 had a related patch set uploaded (by Ottomata):
Deploy eventlogging code to stat1002

https://gerrit.wikimedia.org/r/238963

Change 238963 merged by Ottomata:
Deploy eventlogging code to stat1002

https://gerrit.wikimedia.org/r/238963

@ori, btw, I think @Krinkle can do what he needs now from stat1002 instead of tin.

Krinkle renamed this task from Support kafka in eventlogging client on tin.eqiad.wmnet to Support kafka in eventlogging client on terbium.Sep 21 2015, 6:11 PM

Change 241969 had a related patch set uploaded (by Ottomata):
Deploy eventlogging to terbium

https://gerrit.wikimedia.org/r/241969

Change 241969 merged by Ottomata:
Deploy eventlogging to terbium

https://gerrit.wikimedia.org/r/241969

The eventlogging module now exists on terbium, but kafka is still missing. I'm not using Kafka in client code, but the eventlogging module itself uses it internally, so it's essentially missing a dependency.

import sys
reload(sys)
sys.path.append('/srv/deployment/eventlogging/EventLogging/server')
sys.setdefaultencoding('utf-8')

import eventlogging
events = eventlogging.connect('tcp://eventlog1001.eqiad.wmnet:8600')

for meta in events.filter(schema='NavigationTiming'):
        print meta
        sys.exit()
$ python event-NavigationTiming.py 
Traceback (most recent call last):
  File "event-NavigationTiming.py", line 6, in <module>
    import eventlogging
  File "/srv/deployment/eventlogging/EventLogging/server/eventlogging/__init__.py", line 21, in <module>
    from .handlers import *
  File "/srv/deployment/eventlogging/EventLogging/server/eventlogging/handlers.py", line 26, in <module>
    from pykafka import KafkaClient as PyKafkaClient
ImportError: No module named pykafka
Ottomata closed this task as Resolved.EditedFeb 25 2016, 10:22 PM

This works on terbium and hafnium now:

export PYTHONPATH=/srv/deployment/eventlogging/eventlogging
python /srv/deployment/eventlogging/eventlogging/bin/eventlogging-consumer 'kafka:///kafka1012.eqiad.wmnet:9092?topic=eventlogging_NavigationTiming&zookeeper_connect=conf1001.eqiad.wmnet:2181/kafka/eqiad' stdout://
Dzahn subscribed.

re-opening to ask if this should be generally on all mw maintenance hosts or not. because we are currently setting up the equivalent of terbium in codfw, wasat.

I would like to move this out of the node level and put it in role mediawiki::maintenance if this is part of being an MW maintenance server.

I would like to avoid including stuff straight in site.pp on nodes/hostnames.

Change 279643 had a related patch set uploaded (by Dzahn):
mw:maintenance: move eventlogging from node to role

https://gerrit.wikimedia.org/r/279643

Change 279643 merged by Dzahn:
mw:maintenance: move eventlogging from node to role

https://gerrit.wikimedia.org/r/279643

This is now also installed on mw2090 and will be installed on wasat and future maintenance servers (via the role, not hostnames). Good?