Also related: https://phabricator.wikimedia.org/T110903

In T131977, @Ottomata wrote:

Currently hafnium runs /srv/webperf/ve.py which includes eventlogging from a global python setup.py install. Instead, the ve.service systemd file should put the eventlogging codebase (currently at /srv/deployment/eventlogging/eventlogging, to be moved to /usr/local/src/eventlogging) in the PYTHONPATH env var.

Local testing shows that the global and local install are compatible. Verified via python repl, using import eventlogging help(eventlogging). Although PYTHONPATH must be /srv/deployment/eventlogging/eventlogging (2x) instead of /srv/deployment/eventlogging/eventlogging (3x). Otherwise it imports the global one still.

The two versions also seem compatible, so this should be an easy patch.

Two questions though:

• The puppet config for ve.py (uses global eventlogging) and navtiming.py (uses zmq) are both configured to use tcp://${eventlogging_host}:8600 as their endpoint. I was under the impression that the built-in eventlogging consumer no longer uses zmq, but given both use the same endpoint, it looks like eventlogging.utils.EventConsumer also uses zmq. Is that true?
• When changing the systemd code for ve.pp, I want to make sure that the puppet code still works on a fresh provision. The only declared dependency is the ::eventlogging class in puppet, which I assume provided the global install. In which case we need to add a proper dependency that ensures the local install exists. However, looking closer I see that that requirement uses trebuchet and /srv/deployment, so I guess that's fine as-is. And rather, it's the current global install that wasn't puppetized?

Change 341723 had a related patch set uploaded (by Krinkle):
[operations/puppet] webperf: Use trebuchet install of eventlogging for ve.py

https://gerrit.wikimedia.org/r/341723

The puppet config for ve.py (uses global eventlogging) and navtiming.py (uses zmq) are both configured to use tcp://${eventlogging_host}:8600 as their endpoint. I was under the impression that the built-in eventlogging consumer no longer uses zmq, but given both use the same >endpoint, it looks like eventlogging.utils.EventConsumer also uses zmq. Is that true?

The "official" consumer for EL in eventlog1001 is configured to use kafka but it can be configured to consume from different places as you know, zmq is one of them. See:

nuria@eventlog1001:~$ more /etc/eventlogging.d/consumers/client-side-events-log
kafka:///kafka1012.eqiad.wmnet:9092,kafka1013.eqiad.wmnet:9092,kafka1014.eqiad.wmnet:9092,kafka1018.eqiad.wmnet:9092,kafka1020.eqiad.wmnet:9092,kafka1022.eqiad.wmnet:9092
?topic=eventlogging-client-side&raw=True&api_version=0.9&identity=eventlogging_consumer_files_00

We maintain a Kafka -> ZMQ endpoint just for these processes :) If we can get them all onto Kafka, then we can turn off the ZMQ service.

https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/eventlogging/analytics/zeromq.pp

Although PYTHONPATH must be /srv/deployment/eventlogging

T118772 should probably be done before this. This would make webperf have their own deployment of eventlogging that would not be affected by analytics or eventbus deployments. This would deploy eventlogging from tin to /srv/deployment/eventlogging/webperf (or something like that). (Analytics uses /srv/deployment/eventlogging/analytics).

I'll post over on T118772 with some more info on how this is done.

Change 341723 merged by Ottomata:
[operations/puppet] webperf: Use trebuchet install of eventlogging for ve.py

https://gerrit.wikimedia.org/r/341723