pdns auth metrics unreachable on prod network
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Oct 9 2023, 9:39 AM

Description

While investigating T336854 it was noticed that cloudmetrics hosts try talking to cloudservices100[56]:8081 and fail. It looks like pdns auth listens only on the cloud vps address and not the production address:

cloudservices1005:~# lsof -i tcp:8081
COMMAND      PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
pdns_serv 796099 pdns    7u  IPv4 157511252      0t0  TCP cloudservices1005.private.eqiad.wikimedia.cloud:tproxy (LISTEN)
root@cloudservices1005:~# curl cloudservices1005.private.eqiad.wikimedia.cloud:8081/metrics -s | wc -l
255
root@cloudservices1005:~# curl cloudservices1005.eqiad.wmnet:8081/metrics -s | wc -l
0

There's of course a bunch of solutions here, what do you think @taavi ?

Details

	Subject	Repo	Branch	Lines +/-
	P:openstack::pdns::auth: make pdns web server listen on all IPs	operations/puppet	production	+29 -49

Customize query in gerrit

Related Objects

Mentioned In: T336854: Move labs/wmcs (OpenStack) Prometheus instance off cloudmetrics hosts to prometheus* hosts
Mentioned Here: T347148: Determine how to monitor services in cloud-private / cloudlb
T336854: Move labs/wmcs (OpenStack) Prometheus instance off cloudmetrics hosts to prometheus* hosts

Event Timeline

fgiunchedi created this task.Oct 9 2023, 9:39 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 9 2023, 9:39 AM

It seems like the pdns web server can't listen on multiple interfaces, and we need it on the cloud-private address for designate to do DNS updates. I can see a few possible solutions here:

Use socat or similar on the host to proxy traffic from the prod-private (10.x) address to the cloud-private address
Collect the metrics from some other host that's connected to cloud-private
Collect the metrics from something in Cloud VPS

See also T347148, which is about deciding a more general strategy for this instead of deciding it case-by-case.

Thank you, that's a bummer re: pdns not listening on multiple interfaces.

I don't feel strongly about either implementing something like socat you mentioned or move pdns auth monitoring to Prometheus running in metricsinfra.

The mental model I've been using whether to decide between prod and metricsinfra is whether the service to be monitored is essentially available to cloud-vps or not (FWIW)

fgiunchedi mentioned this in T336854: Move labs/wmcs (OpenStack) Prometheus instance off cloudmetrics hosts to prometheus* hosts.Oct 9 2023, 12:56 PM

fgiunchedi closed this task as Resolved.Oct 10 2023, 8:17 AM

fgiunchedi claimed this task.

Change 966494 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack::pdns::auth: make pdns web server listen on all IPs

https://gerrit.wikimedia.org/r/966494

gerritbot added a project: Patch-For-Review.Oct 17 2023, 9:06 AM

Change 966494 merged by Majavah:

[operations/puppet@production] P:openstack::pdns::auth: make pdns web server listen on all IPs

https://gerrit.wikimedia.org/r/966494

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2023, 11:10 AM