Page MenuHomePhabricator
Create Task
Maniphest T354424

HAProxy 2.6.16/2.8.5 CPU spikes on cp3066
Closed, ResolvedPublic
Actions

Description

Reported to upstream on https://github.com/haproxy/haproxy/issues/2403.

issue appeared with 2.6.15 on December and it's still happening after upgrading to 2.6.16:

image.png (500×1 px, 110 KB)

https://grafana.wikimedia.org/goto/NWOkjTKIk?orgId=1

Details

SubjectRepoBranchLines +/-
hiera: Install HAProxy 2.8 on cp3066operations/puppetproduction+1 -0
aptrepo,haproxy: Allow installing HAProxy 2.8 on bullseyeoperations/puppetproduction+12 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedVgutierrezT354424 HAProxy 2.6.16/2.8.5 CPU spikes on cp3066
 ResolvedVgutierrezT362063 Upgrade to HAProxy 2.6.17

Event Timeline

Vgutierrez created this task.Jan 5 2024, 3:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2024, 3:46 PM
Vgutierrez triaged this task as Medium priority.Jan 5 2024, 3:46 PM
Vgutierrez moved this task from Backlog to Traffic team actively servicing on the Traffic board.
Vgutierrez added a project: Upstream.
• NoOnEtHeMaStA added a commit: rMEXTf658c3375869: Update git submodules.Jan 7 2024, 9:15 PM
Aklapper removed a commit: rMEXTf658c3375869: Update git submodules.Jan 7 2024, 9:25 PM
gerritbot added a comment.Jan 22 2024, 10:42 AM

Change 992097 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] aptrepo,haproxy: Allow installing HAProxy 2.8 on bullseye

https://gerrit.wikimedia.org/r/992097

gerritbot added a project: Patch-For-Review.Jan 22 2024, 10:42 AM
Vgutierrez added a comment.Jan 22 2024, 10:48 AM

as suggested by Willy Tarreau on https://github.com/haproxy/haproxy/issues/2403#issuecomment-1900111538 this issue could be easier to debug on HAProxy 2.8

gerritbot added a comment.Jan 22 2024, 10:58 AM

Change 992098 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Install HAProxy 2.8 on cp3066

https://gerrit.wikimedia.org/r/992098

gerritbot added a comment.Jan 22 2024, 11:01 AM

Change 992097 merged by Vgutierrez:

[operations/puppet@production] aptrepo,haproxy: Allow installing HAProxy 2.8 on bullseye

https://gerrit.wikimedia.org/r/992097

gerritbot added a comment.Jan 22 2024, 11:40 AM

Change 992098 merged by Vgutierrez:

[operations/puppet@production] hiera: Install HAProxy 2.8 on cp3066

https://gerrit.wikimedia.org/r/992098

Stashbot added a comment.Jan 22 2024, 11:41 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-22T11:41:14Z] <vgutierrez> update to HAProxy 2.8.5 on cp3066 - T354424

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 12:30 PM
Stashbot added a comment.Jan 22 2024, 4:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-22T16:53:05Z] <vgutierrez> testing HAProxy tune.bufsize = 32768 in cp3066 - T354424

Stashbot added a comment.Jan 22 2024, 5:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-22T17:05:24Z] <vgutierrez> restore HAProxy tune.bufsize = 16684 in cp3066 - T354424

Vgutierrez renamed this task from HAProxy 2.6.16 CPU spikes on cp3066 to HAProxy 2.6.16/2.8.5 CPU spikes on cp3066.Jan 23 2024, 3:40 PM
Stashbot added a comment.Jan 24 2024, 9:58 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-24T09:58:08Z] <vgutierrez> depooling cp3066 - T354424

Stashbot added a comment.Jan 24 2024, 10:00 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-24T10:00:38Z] <vgutierrez> repool cp3066 - T354424

Stashbot added a comment.Jan 24 2024, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-24T11:31:18Z] <vgutierrez> depooling cp3066 - T354424

Stashbot added a comment.Jan 24 2024, 11:33 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-24T11:33:05Z] <vgutierrez> repool cp3066 - T354424

MoritzMuehlenhoff subscribed.Jan 24 2024, 12:58 PM
Stashbot added a comment.Jan 24 2024, 3:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-24T15:50:08Z] <vgutierrez> disable puppet on cp3066 - T354424

Stashbot added a comment.Jan 24 2024, 4:56 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-24T16:56:37Z] <vgutierrez> enable puppet on cp3066 - T354424

Vgutierrez added a comment.Jan 31 2024, 5:12 PM

Fix already released on HAProxy 2.9: https://www.mail-archive.com/haproxy@formilux.org/msg44547.html

jhathaway subscribed.Jan 31 2024, 5:26 PM
Vgutierrez mentioned this in T362063: Upgrade to HAProxy 2.6.17.Mon, Apr 8, 12:24 PM
Vgutierrez closed this task as Resolved.Mon, Apr 15, 1:40 PM
Vgutierrez claimed this task.
Vgutierrez closed subtask T362063: Upgrade to HAProxy 2.6.17 as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL