[promethus,haproxy] Move to haproxy internal metrics from haproxy_exporter
Open, HighPublic
Actions

Assigned To

Authored By

	dcaro
	Aug 9 2023, 12:58 PM

Description

Currently we have an haproxy exporter that we scrape with prometheus to extract haproxy stats, but since haproxy 2.0.0 it ships with it's own internal metrics endpoint:

See https://github.com/prometheus/haproxy_exporter for details

We should move to that internal endpoint instead and avoid the extra process.

As the stats are named differently, a proposed action plan is:

move all the current users of haproxy_exporter to also collect haproxy stats
- Thumbor is moving itself soon, so no patches needed there
- Toolforge elasticsearch uses haproxy<2, so no stats, might want to upgrade first, but needs changing the config as the current one does not work with 2.1
notify teams that the stats changed
remove the scraping of all the haproxy_exporter and set profile::haproxy_exporter::enable to absent to uninstall/stop
cleanup the haproxy_exporter entries once puppet ran everywhere

Note that we would be collecting duplicated stats for some time until we move everyone to the haproxy internal metrics.

Details

Subject	Repo	Branch	Lines +/-
dns::dotls: expose and gather haproxy internal metrics	operations/puppet	production	+26 -2
cloudlb: move to wmcs prometheus	operations/puppet	production	+16 -30
openstack: use the haproxy internal stat for alerts	operations/alerts	master	+212 -9
cloudlb: allow access to haproxy stats from prometheus	operations/puppet	production	+13 -0
prometheus: fix typo in job name	operations/puppet	production	+1 -1
prometheus: gather stats from haproxy for openstack and cloudlb	operations/puppet	production	+38 -4
thumbor: expose and fetch metrics from haproxy internal endpoint	operations/puppet	production	+42 -3
haproxy_exporter: allow setting as absent	operations/puppet	production	+14 -4

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		dcaro	T343872 [cloudvps] grafana stats for haproxy response time give different data on refresh
		Open		dcaro	T343885 [promethus,haproxy] Move to haproxy internal metrics from haproxy_exporter

Event Timeline

dcaro triaged this task as High priority.Aug 9 2023, 12:58 PM

dcaro created this task.

dcaro updated the task description. (Show Details)Aug 9 2023, 12:59 PM

dcaro added a subtask: T343872: [cloudvps] grafana stats for haproxy response time give different data on refresh.

dcaro removed a subtask: T343872: [cloudvps] grafana stats for haproxy response time give different data on refresh.

dcaro added a parent task: T343872: [cloudvps] grafana stats for haproxy response time give different data on refresh.

Change 947353 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] haproxy_exporter: allow setting as absent

https://gerrit.wikimedia.org/r/947353

Change 947354 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] prometheus: gather stats from haproxy for openstack and cloudlb

https://gerrit.wikimedia.org/r/947354

dcaro updated the task description. (Show Details)Aug 9 2023, 1:42 PM

dcaro updated the task description. (Show Details)

dcaro added a subscriber: fgiunchedi.

fnegri subscribed.Aug 9 2023, 3:08 PM

Change 947353 merged by David Caro:

[operations/puppet@production] haproxy_exporter: allow setting as absent

https://gerrit.wikimedia.org/r/947353

moved comment to T343872: [cloudvps] grafana stats for haproxy response time give different data on refresh --

@dcaro that last comment should be moved to T343872 :)

In T343885#9082900, @fnegri wrote:

@dcaro that last comment should be moved to T343872 :)

yep xd

Change 947354 merged by David Caro:

[operations/puppet@production] prometheus: gather stats from haproxy for openstack and cloudlb

https://gerrit.wikimedia.org/r/947354

We'll also need to allow tcp port 9900 between prometheus and cloudlb/openstack hosts, I just noticed prometheus can't talk to those:

prometheus1005:~$ curl cloudlb1001.eqiad.wmnet:9900/metrics -v
* Uses proxy env variable no_proxy == '.wmnet'
*   Trying 2620:0:861:11f:10:64:151:2:9900...
*   Trying 10.64.151.2:9900...

Change 948083 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] prometheus: fix typo in job name

https://gerrit.wikimedia.org/r/948083

Change 948084 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] cloudlb: allow access to haproxy stats from prometheus

https://gerrit.wikimedia.org/r/948084

Change 948087 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] dns::dotls: expose and gather haproxy internal metrics

https://gerrit.wikimedia.org/r/948087

Change 948092 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] thumbor: expose and fetch metrics from haproxy internal endpoint

https://gerrit.wikimedia.org/r/948092

Change 948092 abandoned by David Caro:

[operations/puppet@production] thumbor: expose and fetch metrics from haproxy internal endpoint

Reason:

superseded by htps://gerrit.wikimedia.org/r/c/operations/puppet/+/946951

https://gerrit.wikimedia.org/r/948092

dcaro updated the task description. (Show Details)Aug 11 2023, 10:08 AM

dcaro updated the task description. (Show Details)Aug 11 2023, 10:10 AM

Change 948098 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/alerts@master] openstack: use the haproxy internal stat for alerts

https://gerrit.wikimedia.org/r/948098

Change 948083 merged by David Caro:

[operations/puppet@production] prometheus: fix typo in job name

https://gerrit.wikimedia.org/r/948083

Change 948084 merged by David Caro:

[operations/puppet@production] cloudlb: allow access to haproxy stats from prometheus

https://gerrit.wikimedia.org/r/948084

Change 948104 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] cloudlb: move to wmcs prometheus

https://gerrit.wikimedia.org/r/948104

In T343885#9083602, @fgiunchedi wrote:

We'll also need to allow tcp port 9900 between prometheus and cloudlb/openstack hosts, I just noticed prometheus can't talk to those:

I wasn't clear enough on this point: we need to open this port on the production/cloud firewall on cr devices

Change 948098 merged by David Caro:

[operations/alerts@master] openstack: use the haproxy internal stat for alerts

https://gerrit.wikimedia.org/r/948098

In T343885#9089323, @fgiunchedi wrote:

In T343885#9083602, @fgiunchedi wrote:

We'll also need to allow tcp port 9900 between prometheus and cloudlb/openstack hosts, I just noticed prometheus can't talk to those:

I wasn't clear enough on this point: we need to open this port on the production/cloud firewall on cr devices

FTR I just noticed other metric collections are affected in this case (e.g. bird):

prometheus1005:~$ curl -v cloudlb1001.eqiad.wmnet:9324/metrics
* Uses proxy env variable no_proxy == '.wmnet'
*   Trying 2620:0:861:11f:10:64:151:2:9324...
*   Trying 10.64.151.2:9324...
^C

I am guessing we can allow all ports from prometheus hosts ACLs on the cr devices (so we don't have to individually allow ports, this matches the production ferm configuration where prometheus hosts can access any ports)

In T343885#9119388, @fgiunchedi wrote:

I am guessing we can allow all ports from prometheus hosts ACLs on the cr devices (so we don't have to individually allow ports, this matches the production ferm configuration where prometheus hosts can access any ports)

FTR I've moved the haproxy jobs to the cloud prometheus instance in https://gerrit.wikimedia.org/r/c/operations/puppet/+/954001 so that solves the specific needs for this task (although we need to fix the cr firewall issue as a part of T336854 anyways).

taavi mentioned this in T336854: Move labs/wmcs (OpenStack) Prometheus instance off cloudmetrics hosts to prometheus* hosts.Aug 31 2023, 12:29 PM

Change 948104 abandoned by David Caro:

[operations/puppet@production] cloudlb: move to wmcs prometheus

Reason:

https://gerrit.wikimedia.org/r/948104

fgiunchedi mentioned this in T349801: Linting problems found for OpenstackAPIResponse.Oct 27 2023, 6:51 AM

fnegri edited projects, added cloud-services-team (FY2023/2024-Q3-Q4); removed cloud-services-team (FY2023/2024-Q1-Q2).Feb 1 2024, 11:14 AM