Page MenuHomePhabricator

Stop using global eventlogging install on hafnium (and any other eventlogging lib user)
Closed, ResolvedPublic13 Estimated Story Points

Description

Currently hafnium runs /srv/webperf/ve.py which includes eventlogging from a global python setup.py install. Instead, the ve.service systemd file should put the eventlogging codebase (currently at /srv/deployment/eventlogging/eventlogging, to be moved to /usr/local/src/eventlogging) in the PYTHONPATH env var.

Event Timeline

Ottomata created this task.Apr 6 2016, 9:29 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 6 2016, 9:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ottomata reassigned this task from Ottomata to Krinkle.Apr 8 2016, 3:03 PM

T110903 is about changing EL consumers to use Kafka. This ticket is about changing the way they load the eventlogging codebase.

  • Change eventlogging webperf puppet so that $eventlogging_path is configurable
  • set Environment="PYTHONPATH=<%= @eventlogging_path %>" in systemd service files.
  • Deploy eventlogging via scap3 or git::clone (otto will do this part).
  • set $eventlogging_path to new deployed location.
JAllemandou moved this task from Incoming to Radar on the Analytics board.
Krinkle reassigned this task from Krinkle to ori.Jul 16 2016, 1:27 AM
Krinkle added a project: Performance-Team.

(Re-assigning per last weeks' perf-team meeting.)

Gilles lowered the priority of this task from Medium to Low.Dec 15 2016, 10:23 PM
Ottomata reassigned this task from ori to Krinkle.Feb 23 2017, 7:40 PM
Ottomata added a subscriber: ori.

Assigning to Timo instead of Ori. Feel free to unassign this or reassign as you see fit. :)

Krinkle added a comment.EditedMar 8 2017, 12:52 AM
In T131977, @Ottomata wrote:

Currently hafnium runs /srv/webperf/ve.py which includes eventlogging from a global python setup.py install. Instead, the ve.service systemd file should put the eventlogging codebase (currently at /srv/deployment/eventlogging/eventlogging, to be moved to /usr/local/src/eventlogging) in the PYTHONPATH env var.

Local testing shows that the global and local install are compatible. Verified via python repl, using import eventlogging help(eventlogging). Although PYTHONPATH must be /srv/deployment/eventlogging/eventlogging (2x) instead of /srv/deployment/eventlogging/eventlogging (3x). Otherwise it imports the global one still.

The two versions also seem compatible, so this should be an easy patch.

Two questions though:

  • The puppet config for ve.py (uses global eventlogging) and navtiming.py (uses zmq) are both configured to use tcp://${eventlogging_host}:8600 as their endpoint. I was under the impression that the built-in eventlogging consumer no longer uses zmq, but given both use the same endpoint, it looks like eventlogging.utils.EventConsumer also uses zmq. Is that true?
  • When changing the systemd code for ve.pp, I want to make sure that the puppet code still works on a fresh provision. The only declared dependency is the ::eventlogging class in puppet, which I assume provided the global install. In which case we need to add a proper dependency that ensures the local install exists. However, looking closer I see that that requirement uses trebuchet and /srv/deployment, so I guess that's fine as-is. And rather, it's the current global install that wasn't puppetized?

Change 341723 had a related patch set uploaded (by Krinkle):
[operations/puppet] webperf: Use trebuchet install of eventlogging for ve.py

https://gerrit.wikimedia.org/r/341723

Nuria added a subscriber: Nuria.Mar 8 2017, 4:38 AM

The puppet config for ve.py (uses global eventlogging) and navtiming.py (uses zmq) are both configured to use tcp://${eventlogging_host}:8600 as their endpoint. I was under the impression that the built-in eventlogging consumer no longer uses zmq, but given both use the same >endpoint, it looks like eventlogging.utils.EventConsumer also uses zmq. Is that true?

The "official" consumer for EL in eventlog1001 is configured to use kafka but it can be configured to consume from different places as you know, zmq is one of them. See:

nuria@eventlog1001:~$ more /etc/eventlogging.d/consumers/client-side-events-log
kafka:///kafka1012.eqiad.wmnet:9092,kafka1013.eqiad.wmnet:9092,kafka1014.eqiad.wmnet:9092,kafka1018.eqiad.wmnet:9092,kafka1020.eqiad.wmnet:9092,kafka1022.eqiad.wmnet:9092
?topic=eventlogging-client-side&raw=True&api_version=0.9&identity=eventlogging_consumer_files_00

We maintain a Kafka -> ZMQ endpoint just for these processes :) If we can get them all onto Kafka, then we can turn off the ZMQ service.

https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/eventlogging/analytics/zeromq.pp

Although PYTHONPATH must be /srv/deployment/eventlogging

T118772 should probably be done before this. This would make webperf have their own deployment of eventlogging that would not be affected by analytics or eventbus deployments. This would deploy eventlogging from tin to /srv/deployment/eventlogging/webperf (or something like that). (Analytics uses /srv/deployment/eventlogging/analytics).

I'll post over on T118772 with some more info on how this is done.

Change 341723 merged by Ottomata:
[operations/puppet] webperf: Use trebuchet install of eventlogging for ve.py

https://gerrit.wikimedia.org/r/341723

Krinkle closed this task as Resolved.Mar 15 2017, 2:05 AM
Krinkle removed a project: Patch-For-Review.
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM