I'm wondering if reducing the hard-stop-after window from 5m to something smaller than the scrap time from prometheus (once a minute) could get rid of this. What are your thoughts @fgiunchedi?

I've wanted to get reduce the hard-stop-after value for some time already and this seems like a good excuse to do it :)

I'm not familiar with haproxy reload architecture, however your theory seems sound to me @Vgutierrez!

Regarding HAProxy reload process, basically HAProxy spawns a new process and hands over all the file descriptors to the new process (that's been started with the new configuration)

@fgiunchedi checking haproxy_frontend_connections_total{proxy="stats"} values at https://grafana.wikimedia.org/goto/sd52V8q4k?orgId=1 it looks like the scraper doesn't close the connection ever, so if it is ok with you I'd like to trigger a change in that behavior by adding a Connection: close header

Change 943506 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] haproxy: Disable KA on stats frontend

https://gerrit.wikimedia.org/r/943506

Change 943506 merged by Vgutierrez:

[operations/puppet@production] haproxy: Disable KA on stats frontend

https://gerrit.wikimedia.org/r/943506

It looks like it's a matter of how we graph the data, please see: https://grafana.wikimedia.org/goto/7xCydjqVk?orgId=1

first panel is the original one using rate()[5m], as soon as we reduce the period to 2m or we switch to irate the issue dissapears.

Looking at the raw data https://w.wiki/7AxF there's indeed a counter "reset" e.g. around 16:00

I'm not sure offhand why moving to a smaller period fixes things, but either solution I think is fine @Vgutierrez

I'm not sure that a smaller period does fix things. Attached is a 5m and 2m. Switching to irate() is showing similar things, too.

It can be observed on @Vgutierrez 's dashboard as well:

https://grafana.wikimedia.org/d/rkL2dCq4z/vgutierrez-haproxy-rps-test?orgId=1&from=1692882998791&to=1692888696352

Can someone help me double-check if this is still a problem? I don't see it in the dashboards above, selecting a more recent time interval.

yes, it's still hapenning https://grafana.wikimedia.org/goto/SHdP6s3HR?orgId=1:

I believe this will be fixed when we upgrade to HAProxy 3.0 given it provides persistent stats.

In T343000#11241334, @Vgutierrez wrote:

yes, it's still hapenning https://grafana.wikimedia.org/goto/SHdP6s3HR?orgId=1:

I believe this will be fixed when we upgrade to HAProxy 3.0 given it provides persistent stats.

Oh I see, I should have zoomed in a bit more. OK thanks, leaving it open.

The behavior reported in this ticket is still happening; hourly reloads correspond to hourly dips in haproxy_frontend_http_requests_total, for example.

Persistent stats are available since HAProxy 3.0 and we appear to be running 3.2.15, so that's something we could try to use; stats can be persisted to disk or memory, but the latter is experimental and only supported in 3.3.

There's one potential annoyance to implement this: we have to assign a unique GUID to each frontend, backend, listen, and server object in the configuration; a naive way to assign these GUIDs could be to have node-specific attributes in hiera for each instance of these objects. Not sure if there's a better way. since it doesn't seem that these GUIDs have to be unique across the fleet, we can probably just hardcode them in the configuration

Aside from that, we need to give HAProxy a directory to read the stats file from, set the path of this file in the configuration and modify the systemd service to execute a dump of the stats before stop and reload.

Change #1343039 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] [WIP] cache::haproxy: implement optional persistent stats

https://gerrit.wikimedia.org/r/1343039

I did some local testing to validate the approach of my CR and it seems to be working as intended; metrics like haproxy_frontend_http_requests_total or haproxy_server_bytes_in_total are persisted across reloads (SIGUSR2) for any object with a "guid" assigned and objects without guid have their stats reset as expected.

To validate that the stats file is actually used I also did a test where I did a few requests, run the dump stats-file admin command, reloaded haproxy, did a few more requests and reloaded again without also re-running the dump command; this resulted in the metrics being reset to the previous value. In other words:

• start process; haproxy_frontend_http_requests_total{proxy="web"} 0
• do four requests
• haproxy_frontend_http_requests_total{proxy="web"} 4
• dump stats file
• reload
• haproxy_frontend_http_requests_total{proxy="web"} 4
• do three more requests
• haproxy_frontend_http_requests_total{proxy="web"} 7
• reload (without dump stats file)
• haproxy_frontend_http_requests_total{proxy="web"} 4

Couple more notes:

• If the stats file is not present at startup time, haproxy will print a warning; in my test environment it's just a harmless warning, but we should ensure that the production configuration won't treat it as an error
• Given that the whole persistence thing is orchestrated by systemd, there's a small gap between the "dump stats" command being issued and the reload signal being sent; this result in some minor loss of metrics (~14% in my test). An improvement over the current situation, but still not perfect :)

@Vgutierrez brought up an interesting point: after a crash of HAProxy, the new instance will load a stale stats file which will result in weird metrics being reported; having inaccurate metrics from one or more nodes would be risky. The longer HAProxy goes without a reload, the weirder the metrics would get after a crash.

I see a few options to mitigate the impact of this scenario:

1. the stats dump command is executed by systemd pre-reload and pre-restart, and by a cron job every X minutes; the problem with this approach is that now we need to handle potential concurrent writes of the stats file, meaning we need to use a lock file. I also don't like the idea of restarts and reloads being delayed because the lock is being held. Small loss of metrics but added complexity.
2. the stats dump command is only executed by systemd (which is the initial implementation in my CR); in this case the tradeoff is stale/weird metrics (up to one hour, assuming hourly reloads). Potentially large loss of metrics, low complexity.
3. the stats dump command is only executed by a cron job, like every 1 or 2 minutes; in this case we will observe small 1/2 minute gaps in metrics during reloads and crashes. Small loss of metrics, low complexity.

Considering that each of the evaluated options comes with at least a small loss of metrics during certain events, it seems that option 3 is the better one since it also covers the crash scenario.

Change #1344222 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable HAProxy's persistent stats on two magru nodes

https://gerrit.wikimedia.org/r/1344222

Change #1343039 merged by Daniel Kertesz:

[operations/puppet@production] cache::haproxy: implement support for persistent stats

https://gerrit.wikimedia.org/r/1343039

Mentioned in SAL (#wikimedia-operations) [2026-09-24T14:32:41Z] <dkertesz> depooling cp7001|7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000)

Change #1344222 merged by Daniel Kertesz:

[operations/puppet@production] hiera: enable HAProxy's persistent stats on two magru nodes

https://gerrit.wikimedia.org/r/1344222

Mentioned in SAL (#wikimedia-operations) [2026-09-24T14:51:22Z] <dkertesz> repooling cp7001|7011 after successful testing (T343000)

I have merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343039 which introduces support for persistent stats in haproxy; to test that the CR is working fine and is not having negative impact I've enable the persistent stats flag on two cp nodes in magru: cp7001 and cp7011: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222
The hosts were de-pooled before applying the change (via puppet); haproxy was reloaded as part of the puppet run. After the puppet run finished, I've also restarted haproxy to ensure that it could start successfully. After the tests were done, the two nodes were re-pooled.

Change #1345067 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] cache::haproxy: move the stats file to /run/haproxy

https://gerrit.wikimedia.org/r/1345067

Change #1345100 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable HAProxy's persistent stats on two esams nodes

https://gerrit.wikimedia.org/r/1345100

Change #1345101 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: remove individual entries for haproxy's persistent stats

https://gerrit.wikimedia.org/r/1345101

Change #1345102 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable haproxy's persistent stats in magru

https://gerrit.wikimedia.org/r/1345102

Change #1345103 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable haproxy's persistent stats in drmrs

https://gerrit.wikimedia.org/r/1345103

Change #1345104 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable haproxy's persistent stats in eqsin

https://gerrit.wikimedia.org/r/1345104

Change #1345105 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable haproxy's persistent stats in ulsfo

https://gerrit.wikimedia.org/r/1345105

Change #1345106 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable haproxy's persistent stats in esams

https://gerrit.wikimedia.org/r/1345106

Change #1345107 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable haproxy's persistent stats in codfw

https://gerrit.wikimedia.org/r/1345107

Change #1345108 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] hiera: enable haproxy's persistent stats in eqiad

https://gerrit.wikimedia.org/r/1345108

Change #1345067 merged by Daniel Kertesz:

[operations/puppet@production] cache::haproxy: move the stats file to /run/haproxy

https://gerrit.wikimedia.org/r/1345067

Mentioned in SAL (#wikimedia-operations) [2026-09-25T14:30:14Z] <dkertesz> moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - T343000

Something weird happened: we decided to move the stats file to /run so that it wouldn't survive reboots, so I prepared a CR and applied it as follows on cp7001 and cp7011 (where we're currently testing this change):

sudo disable-puppet "moving haproxy stats file to /run/haproxy - T343000"

# merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1345067
# run puppet-merge

sudo cp /var/lib/haproxy/stats-file /run/haproxy/
sudo run-puppet-agent -e "moving haproxy stats file to /run/haproxy - T343000"
sudo rm /var/lib/haproxy/stats-file

The ideas was that by the time puppet reloaded haproxy, the stats file was already present in the new directory and we would not see a reset of metrics; instead I got this and this.

Even weirder, cp7011 had a similar hallucinated spike around midnight UTC and it kinda seems to coincide with haproxy reloads across the cluster (zooming in on the other upload nodes shows the usual dip that happens after a reload.

There have been other reload events in the past 24 hours (journalctl --since "24 hours ago" -u haproxy | grep -i reload), but it's only this one at midnight on cp7011 that exhibits this weird behavior.

Update: potentially relevant, but a bit vague: https://github.com/haproxy/haproxy/issues/2483#issuecomment-2579845321 mentions "the monitoring collection hallucinate big values"

it definitely looks like you're hitting the same issue mentioned on https://github.com/haproxy/haproxy/issues/2483#issuecomment-2579845321, definitely not great for us either

Change #1345486 had a related patch set uploaded (by Daniel Kertesz; author: Daniel Kertesz):

[operations/puppet@production] Revert "hiera: enable HAProxy's persistent stats on two magru nodes"

https://gerrit.wikimedia.org/r/1345486

Change #1345486 merged by Daniel Kertesz:

[operations/puppet@production] Revert "hiera: enable HAProxy's persistent stats on two magru nodes"

https://gerrit.wikimedia.org/r/1345486

Reverted the change that enabled persistent stats on cp7001 and cp7011 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1345486
There were more hallucinated metrics over the weekend reporting extremely high bogus values.

We might give this another try when we upgrade to 3.4, which marks the SHM version of the stats file as no longer experimental:

MINOR: config: shm-stats-file is no longer experimental (in haproxy 3.4-dev14)

Reference: https://github.com/haproxy/haproxy/blob/master/CHANGELOG