Page MenuHomePhabricator

Support kafka in eventlogging client on terbium
Closed, ResolvedPublic

Description

The eventlogging client python module installed on tin.eqiad.wmnet doesn't support Kafka.

Which more and more stuff moving away from ZMQ, this makes it impossible to test or debug event streams. Not unless there is another host I can access that does have a working eventlogging-kafka client.

It should probably also be installed on terbium instead of tin to discourage running of such processes on the deployment host.


Event Timeline

Krinkle created this task.Sep 15 2015, 3:30 PM
Krinkle raised the priority of this task from to High.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 15 2015, 3:30 PM
Krinkle set Security to None.

I don't think we even knew that eventlogging was deployed to tin. Both terbium and tin are precise. Is there a Trusty host where we can install this?

We should install it on stat1002.

Krinkle updated the task description. (Show Details)Sep 16 2015, 8:41 PM

Change 238963 had a related patch set uploaded (by Ottomata):
Deploy eventlogging code to stat1002

https://gerrit.wikimedia.org/r/238963

Change 238963 merged by Ottomata:
Deploy eventlogging code to stat1002

https://gerrit.wikimedia.org/r/238963

ori claimed this task.Sep 21 2015, 6:01 PM

@ori, btw, I think @Krinkle can do what he needs now from stat1002 instead of tin.

Krinkle renamed this task from Support kafka in eventlogging client on tin.eqiad.wmnet to Support kafka in eventlogging client on terbium.Sep 21 2015, 6:11 PM

Change 241969 had a related patch set uploaded (by Ottomata):
Deploy eventlogging to terbium

https://gerrit.wikimedia.org/r/241969

Change 241969 merged by Ottomata:
Deploy eventlogging to terbium

https://gerrit.wikimedia.org/r/241969

Krinkle added a comment.EditedOct 6 2015, 9:17 PM

The eventlogging module now exists on terbium, but kafka is still missing. I'm not using Kafka in client code, but the eventlogging module itself uses it internally, so it's essentially missing a dependency.

import sys
reload(sys)
sys.path.append('/srv/deployment/eventlogging/EventLogging/server')
sys.setdefaultencoding('utf-8')

import eventlogging
events = eventlogging.connect('tcp://eventlog1001.eqiad.wmnet:8600')

for meta in events.filter(schema='NavigationTiming'):
        print meta
        sys.exit()
$ python event-NavigationTiming.py 
Traceback (most recent call last):
  File "event-NavigationTiming.py", line 6, in <module>
    import eventlogging
  File "/srv/deployment/eventlogging/EventLogging/server/eventlogging/__init__.py", line 21, in <module>
    from .handlers import *
  File "/srv/deployment/eventlogging/EventLogging/server/eventlogging/handlers.py", line 26, in <module>
    from pykafka import KafkaClient as PyKafkaClient
ImportError: No module named pykafka
Ottomata closed this task as Resolved.EditedFeb 25 2016, 10:22 PM

This works on terbium and hafnium now:

export PYTHONPATH=/srv/deployment/eventlogging/eventlogging
python /srv/deployment/eventlogging/eventlogging/bin/eventlogging-consumer 'kafka:///kafka1012.eqiad.wmnet:9092?topic=eventlogging_NavigationTiming&zookeeper_connect=conf1001.eqiad.wmnet:2181/kafka/eqiad' stdout://
Dzahn reopened this task as Open.Mar 25 2016, 5:58 PM
Dzahn added a subscriber: Dzahn.

re-opening to ask if this should be generally on all mw maintenance hosts or not. because we are currently setting up the equivalent of terbium in codfw, wasat.

I would like to move this out of the node level and put it in role mediawiki::maintenance if this is part of being an MW maintenance server.

I would like to avoid including stuff straight in site.pp on nodes/hostnames.

Change 279643 had a related patch set uploaded (by Dzahn):
mw:maintenance: move eventlogging from node to role

https://gerrit.wikimedia.org/r/279643

Change 279643 merged by Dzahn:
mw:maintenance: move eventlogging from node to role

https://gerrit.wikimedia.org/r/279643

Dzahn added a comment.Mar 25 2016, 6:23 PM

This is now also installed on mw2090 and will be installed on wasat and future maintenance servers (via the role, not hostnames). Good?

Dzahn closed this task as Resolved.Mar 31 2016, 1:15 AM