Page MenuHomePhabricator

openstack: alert for cloudvirts without aggregate or with unexpected set of them
Open, MediumPublic

Description

There's nothing preventing a cloudvirt from being left over in maintenance (i.e. nova-compute service disabled on it at the compute service level.

With T424802: Revisit cloudvirt maintenance story in place we can now do the following:

  • Turn compute service list -f json into prometheus metrics exporting the per-host service status
  • Alert when an host has nova-compute disabled for longer than a certain threshold
  • And/or alert when an unexpected number of cloudvirts has nova-compute disabled
  • Make CloudVirtDown not page if the host has nova-compute disabled (?) and if there are no known VMs scheduled on it according to nova (?)

The -f json output looks like this:

{
  "ID": "9e43876c-e482-461d-9ae0-ad8d1376e49e",
  "Binary": "nova-compute",
  "Host": "cloudvirt1080",
  "Zone": "nova",
  "Status": "enabled",
  "State": "up",
  "Updated At": "2026-07-09T10:28:27.000000"
}

note this output includes other nova services too running on e.g. cloudcontrol

{
   "ID": "755f82d2-62c2-4525-bbc6-8de6ce0bbb1b",
   "Binary": "nova-conductor",
   "Host": "cloudcontrol1006",
   "Zone": "internal",
   "Status": "enabled",
   "State": "up",
   "Updated At": "2026-07-09T10:28:31.000000"
 },

Event Timeline

fgiunchedi raised the priority of this task from Low to Medium.Jul 9 2026, 10:46 AM
fgiunchedi updated the task description. (Show Details)

Change #1313958 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: export compute service info to prometheus

https://gerrit.wikimedia.org/r/1313958

The patch at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1313958 will export compute service status as prometheus metrics. That will enable us to (for example):

  • alert on hosts left in maintenance for long periods of time (state=up, status=disabled)
  • skip paging on cloudvirt hosts that are purposefully down (state=down, status=disabled)
  • include/exclude hosts in mainteance from cpu/mem capacity dashboards