Page MenuHomePhabricator
Create Task
Maniphest T367389

[k8s,infra,alerting] improve HAproxy and k8s apiserver interaction
Closed, ResolvedPublic
Actions

Description

In incident T367348: Incident: 2024-06-12 toolforge k8s control plane we discovered a failure case where the Kubernetes API server's port is open but it's not responding to any API requests. In that case HAProxy still sees the failed servers as up and tries to send traffic to them. The current HAProxy health check is a simple TCP check, I think it should be replaced by a HTTPS check for the /healthz endpoint.

Also review that we have alerts for the case where the API servers are not healthy according to HAProxy.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
toolforge: haproxy: check the k8s api-server /healthz endpointoperations/puppetproduction+2 -3
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
kubernetes: add some basic HAproxy alertsrepos/cloud/toolforge/alerts!15aborreroarturo-220-kubernetes-add-somemain
Customize query in GitLab

Related Objects
Search...

Event Timeline

• aborrero created this task.Jun 13 2024, 9:34 AM
• aborrero triaged this task as High priority.
• aborrero moved this task from Backlog to Next on the User-aborrero board.
taavi removed taavi as the assignee of this task.Jun 13 2024, 9:39 AM
taavi updated the task description. (Show Details)
taavi updated the task description. (Show Details)
dcaro renamed this task from toolforge: improve HAproxy and k8s apiserver interaction to [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction.Jun 13 2024, 9:46 AM
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.
dcaro subscribed.
dcaro merged a task: T367349: Fix HA proxy load-balancer health check monitor to not poll nodes where the API is not responding.Jun 13 2024, 9:48 AM
dcaro mentioned this in T348633: [api-gateway] add alert for uptime.Jun 13 2024, 10:20 AM
gerritbot added a comment.Jun 18 2024, 3:37 PM

Change #1047113 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] toolforge: haproxy: use HTTP healthcheck for the k8s api-server

https://gerrit.wikimedia.org/r/1047113

gerritbot added a project: Patch-For-Review.Jun 18 2024, 3:37 PM
gerritbot added a comment.Jun 19 2024, 9:51 AM

Change #1047113 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] toolforge: haproxy: check the k8s api-server /healthz endpoint

https://gerrit.wikimedia.org/r/1047113

Maintenance_bot removed a project: Patch-For-Review.Jun 19 2024, 10:30 AM
• aborrero lowered the priority of this task from High to Medium.Jun 19 2024, 11:04 AM
CodeReviewBot added a project: Patch-For-Review.Jun 19 2024, 1:18 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/15

kubernetes: add some basic HAproxy alerts

CodeReviewBot added a comment.Jun 20 2024, 9:04 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/15

kubernetes: add some basic HAproxy alerts

Maintenance_bot removed a project: Patch-For-Review.Jun 20 2024, 9:32 AM
• aborrero closed this task as Resolved.Jun 20 2024, 12:30 PM
• aborrero claimed this task.

Alert confirmed to work as expected:

image.png (313×522 px, 50 KB)

• aborrero mentioned this in T368044: Toolforge: redeploy kyverno after the outage.Jun 20 2024, 12:48 PM
• aborrero removed a subtask: T367956: haproxy: install some command line interface.Jun 21 2024, 11:36 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits