It seems like metricsinfra is now sometimes picking the v6 address to use when scraping dualstack instances, even though the Prometheus hosts are in the legacy VLAN. (Note that Prometheus doesn't seem to be consistent which address it's using...)
Options:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T53494 Use Beta cluster as a true canary for code deployments (epic) | |||
| Declined | None | T87220 Minimize infrastructure differences between Beta Cluster and production | |||
| Open | None | T211677 Support IPv6 in beta | |||
| Open | None | T404466 Support IPv6-only VMs | |||
| Open | None | T392688 Enable IPv6 on Cloud VPS infrastructure services | |||
| Resolved | taavi | T379175 Enable IPv6 for the Cloud VPS web proxy | |||
| Open | None | T270694 CloudVPS: introduce tenant networks | |||
| Resolved | • aborrero | T364725 Migrate Cloud VPS instances to VXLAN based networks | |||
| Open | None | T392509 Enable IPv6 for Toolforge services | |||
| Resolved | taavi | T392506 Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service | |||
| Resolved | taavi | T211575 Enable IPv6 on toolforge.org | |||
| Open | None | T220306 Add IPv6 monitoring | |||
| Resolved | • aborrero | T37947 Enable IPv6 on CloudVPS | |||
| Resolved | taavi | T392570 metricsinfra: Support scraping v6-enabled instances |
This is also an issue on Toolforge. I'm going to use that to prototype a fix and then roll out a proper fix onto metricsinfra.
Change #1139455 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:toolforge: prometheus: Use DNS names to look up scrape targets
Change #1139511 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:wmcs::metricsinfra: Add instance FQDN template
Change #1139455 merged by Majavah:
[operations/puppet@production] P:toolforge: prometheus: Use DNS names to look up scrape targets
Change #1139514 had a related patch set uploaded (by Majavah; author: Majavah):
[cloud/metricsinfra/prometheus-configurator@master] prometheus: Use DNS names to refer to instances
Change #1139511 merged by Majavah:
[operations/puppet@production] P:wmcs::metricsinfra: Add instance FQDN template
Change #1139514 merged by jenkins-bot:
[cloud/metricsinfra/prometheus-configurator@master] prometheus: Use DNS names to refer to instances
Change #1139858 had a related patch set uploaded (by Majavah; author: Majavah):
[cloud/metricsinfra/prometheus-configurator@master] prometheus: Run blackbox probes on IPv4
Change #1139858 merged by jenkins-bot:
[cloud/metricsinfra/prometheus-configurator@master] prometheus: Run blackbox probes on IPv4