Page MenuHomePhabricator
Create Task
Maniphest T431946

2026-07-11 Incident - Toolforge haproxies flapping
Closed, ResolvedPublic
Actions

Description

TODO

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
toolforge::k8s::haproxy: tweak timeouts, limits and open filesDavid Carooperations/puppetproduction+22 -9
P:toolforge::k8s::haproxy: Decrease client timeoutsMajavahoperations/puppetproduction+4 -1
P:toolforge::k8s::haproxy: Set aggressive timeouts on HTTP redirectsMajavahoperations/puppetproduction+7 -0
P:toolforge::k8s::haproxy: Set a global connection limitMajavahoperations/puppetproduction+10 -8
P:toolforge::k8s::haproxy: Increase open file descriptor limitMajavahoperations/puppetproduction+9 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

dcaro created this task.Jul 12 2026, 12:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2026, 12:34 PM
dcaro claimed this task.Jul 12 2026, 12:34 PM
dcaro triaged this task as High priority.
dcaro added a subtask: T431967: Intermittent timeouts on Phab-ban tool.Jul 13 2026, 8:49 AM
gerritbot added a comment.Jul 13 2026, 9:33 AM

Change #1310024 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] toolforge::k8s::haproxy: tweak timeouts, limits and open files

https://gerrit.wikimedia.org/r/1310024

gerritbot added a project: Patch-For-Review.Jul 13 2026, 9:33 AM
dcaro reassigned this task from dcaro to taavi.Jul 13 2026, 9:44 AM
gerritbot added a comment.Jul 13 2026, 10:05 AM

Change #1310028 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Increase open file descriptor limit

https://gerrit.wikimedia.org/r/1310028

gerritbot added a comment.Jul 13 2026, 10:05 AM

Change #1310029 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Set a global connection limit

https://gerrit.wikimedia.org/r/1310029

gerritbot added a comment.Jul 13 2026, 10:06 AM

Change #1310030 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Set aggressive timeouts on HTTP redirects

https://gerrit.wikimedia.org/r/1310030

gerritbot added a comment.Jul 13 2026, 10:06 AM

Change #1310031 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Decrease client timeouts

https://gerrit.wikimedia.org/r/1310031

gerritbot added a comment.Jul 13 2026, 10:10 AM

Change #1310028 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Increase open file descriptor limit

https://gerrit.wikimedia.org/r/1310028

gerritbot added a comment.Jul 13 2026, 10:12 AM

Change #1310029 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Set a global connection limit

https://gerrit.wikimedia.org/r/1310029

gerritbot added a comment.Jul 13 2026, 10:17 AM

Change #1310030 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Set aggressive timeouts on HTTP redirects

https://gerrit.wikimedia.org/r/1310030

gerritbot added a comment.Jul 13 2026, 10:18 AM

Change #1310031 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Decrease client timeouts

https://gerrit.wikimedia.org/r/1310031

taavi added projects: Toolforge, tools-infrastructure-team.Jul 13 2026, 10:50 AM
taavi closed this task as Resolved.Jul 14 2026, 12:59 PM

Optimistically closing

aputhin moved this task from Backlog to Done on the tools-platform-team board.Thu, Jul 16, 10:39 AM
LSobanski added projects: Wikimedia-Incident, Incident Severity 3.Mon, Jul 20, 4:24 PM
gerritbot added a comment.Mon, Jul 20, 4:29 PM

Change #1310024 abandoned by David Caro:

[operations/puppet@production] toolforge::k8s::haproxy: tweak timeouts, limits and open files

Reason:

Superseded by the stack at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1310030

https://gerrit.wikimedia.org/r/1310024

Maintenance_bot removed a project: Patch-For-Review.Mon, Jul 20, 4:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits