We have accidentally triggered alerts by forgetting to depool nodes before a reimage. We might also have nodes failing health checks because they haven't properly joined the cluster, or for any number of reasons. A real-time view of each node's health status would help us triage these issues quickly, hopefully before they generate alerts for other teams.

Creating this ticket to:

Find the best source of this information:
(@Volans pointed me to this alerts dashboard)
@dcausse pointed me to this PyBal dashboard which does almost everything we want
Use the information to create a tool (probably a dashboard) that displays real-time (or as near as is practical) information on node status per pool.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		bking	T392222 Create ops-focused OpenSearch dashboard
		Open		None	T394676 Create tool that displays real-time load balancer health status per pool/node

Event Timeline

bking created this task.May 19 2025, 2:05 PM

bking triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2025, 2:05 PM

bking renamed this task from Create tool that displays real-time load balancer health status per pool/host to Create tool that displays real-time load balancer health status per pool/node.May 19 2025, 2:14 PM

bking updated the task description. (Show Details)

bking added a subscriber: Volans.

bking updated the task description. (Show Details)May 19 2025, 2:43 PM

bking added a subscriber: dcausse.

This should be addressed as part of T392222

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.May 30 2025, 7:47 AM

Gehel edited projects, added: Data-Platform-SRE (2025.05.24 - 2025.06.13); removed: Data-Platform-SRE.May 30 2025, 9:39 AM

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.

Gehel edited projects, added: Data-Platform-SRE (2025.06.13 - 2025.07.04); removed: Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:49 AM

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.

BTullis edited projects, added: Data-Platform-SRE (2025.07.05 - 2025.07.25); removed: Data-Platform-SRE (2025.06.13 - 2025.07.04).Jul 4 2025, 3:59 PM

BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 4 2025, 4:18 PM

BTullis edited projects, added: Data-Platform-SRE (2025.07.26 - 2025.08.15); removed: Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:48 PM

BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.

@gmodena , @Lydia_Pintscher and I have been discussing something similar in Slack .

Config-master's pools.json has the information we need; we just need a convenient way to get it into our dashboards. Long-term, we could do that with the grafana json datasource plugin , but that would have to be approved by Observability. Absent that, we'd probably just link it from a description panel (assuming we can put HTML in descriptions) and/or link from a Wikitech doc.

BTullis edited projects, added: Data-Platform-SRE (2025.08.16 - 2025.09.05); removed: Data-Platform-SRE (2025.07.26 - 2025.08.15).Aug 20 2025, 10:29 AM

BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.

Gehel edited projects, added: Data-Platform-SRE (2025.09.05 - 2025.09.26); removed: Data-Platform-SRE (2025.08.16 - 2025.09.05).Sep 5 2025, 2:24 PM

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.

Gehel added a project: Essential-Work.Sep 11 2025, 12:12 PM

Gehel edited projects, added: Data-Platform-SRE (2025.09.26 - 2025.10.17); removed: Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.

Gehel edited projects, added: Data-Platform-SRE (2025.10.17 - 2025.11.07); removed: Data-Platform-SRE (2025.09.26 - 2025.10.17).Oct 17 2025, 8:50 AM

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.Oct 17 2025, 8:52 AM

Gehel edited projects, added: Data-Platform-SRE (2025.11.07 - 2025.11.28); removed: Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:32 PM

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.

Gehel edited projects, added: Data-Platform-SRE (2026.01.05 - 2026.01.23); removed: Data-Platform-SRE (2025.11.07 - 2025.11.28).Jan 8 2026, 1:15 PM