ATM the hiera key profile::wmcs::services::toolsdb::primary_server must be changed manually as per procedure to promote a replica to primary. Since we switch the DNS name tools.db.svc.wikimedia.cloud anyways as part of the procedure, the hiera key above can be replaced by checking DNS.

A solution outline looks like this (untested)

$service_record = "tools.db.svc.wikimedia.cloud"
$is_active_host = (ipresolve($service_record, 4) == $facts['networking']['ip']) or (ipresolve($service_record, 6) == $facts['networking']['ip6'])

Then $is_active_host can be used to set units as present or absent (or the service as running/stopped)

Incident metadata:

• This was an action item for Incident 2025-11-11 WMCS toolsdb primary down
• Incident document: https://docs.google.com/document/d/1xinOSzx2JtfX_NYC4ntn4lxEC3PViszPcn6JDkojHuQ/edit?tab=t.0