Page MenuHomePhabricator
Create Task
Maniphest T341966

prometheus in codfw can't reach cloudservices' pdns to fetch metrics
Closed, ResolvedPublic
Actions

Description

There are two jobs to fetch powerdns metrics in codfw from prometheus:

root@prometheus2005:/srv/prometheus/ops/targets# cat cloud-dev-pdns_codfw.yaml
# This file is managed by puppet
---
- labels:
    cluster: misc
    site: codfw
  targets:
  - cloudservices2004-dev:8081
  - cloudservices2005-dev:8081

root@prometheus2005:/srv/prometheus/ops/targets# cat cloud-dev-pdns-rec_codfw.yaml
# This file is managed by puppet
---
- labels:
    cluster: misc
    site: codfw
  targets:
  - cloudservices2004-dev:8082
  - cloudservices2005-dev:8082

These have stopped working since a month (JobUnavailable alert is firing). Is this known/expected ?

Details

SubjectRepoBranchLines +/-
CR: cloud-host: allow return traffic for PDNS serversoperations/homer/publicmaster+3 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedaborreroT296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
 ResolvedaborreroT297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
 ResolvedaborreroT324992 cloudlb: create PoC on codfw
 ResolvedaborreroT327908 rename cloudgw2001-dev into cloudlb2001-dev
 ResolvedayounsiT327930 Is Vlan 2122 cloud-support1-b-codfw required?
 ResolvedcmooneyT327919 Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it
 ResolvedPapaulT331470 Run 2x1G links from asw-b1-codfw to cloudsw1-b1-codfw
Restricted Task
 ResolvedcmooneyT333316 Homer unable to commit config to cloudsw1-b1-codfw (QFX5120 21.4R3.16)
 ResolvedcmooneyT336368 cloudgw: review security policy for edge network
 ResolvedaborreroT332153 cloudlb PoC: prepare backends
 ResolvedaborreroT336963 cloudcontrol2001-dev can't reach cloud-vps public IPs
 ResolvedaborreroT335759 cloud-private subnet: introduce new domain
 Resolved• taaviT336566 cloud: failures resolving some `wikimedia.cloud` domains
 OpencmooneyT346428 Netbox: Add support for our complex host network setups in provision script
 ResolvedaborreroT346410 need private IPs for cloudvirt200[4-6]-dev.codfw.wmnet
 OpencmooneyT347375 Netbox device location information not available on the first Puppet run of a device
 ResolvedaborreroT335760 Modify Bird module to allow source IP to be passed to template
 ResolvedaborreroT336071 cloudlb: figure out routing
 ResolvedaborreroT337758 puppet: interface::route resource does not persists settings
 ResolvedaborreroT338936 cloudlb: figure out plans for eqiad1
 ResolvedaborreroT338937 cloudlb: review swift/radosgw status
ResolvedAndrewT341380 Open swift port (28080) to the public internet
 ResolvedAndrewT341484 Horizon object store UI doesn't allow uploading of files
 ResolvedAndrewT341640 Move Horizon deploy to a Docker container
 ResolvedAndrewT341509 radosgw+keystone chokes on projects with '-' in their id
 ResolvedAndrewT343158 Support OpenStack projects where project name != project id
 OpenNoneT366679 openstack-browser support for projects where id != name
 ResolvedAndrewT347927 Better swift error messages for projects with invalid chars
 ResolvedAndrewT366301 Document and enforce restrictions on openstack project names
 ResolvedaborreroT340536 cloud-private: figure out, implement and test cross-DC traffic
 OpenNoneT317177 [tracking] Don't keep on the public vlans hosts that don't require it
 OpenNoneT340446 Cloud VPS Designate setup improvements
 ResolvedaborreroT307357 Move cloud vps ns-recursor IPs to host/row-independent addressing
 ResolvedcmooneyT336587 cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan
 ResolvedfgiunchediT341966 prometheus in codfw can't reach cloudservices' pdns to fetch metrics

Event Timeline

fgiunchedi created this task.Jul 17 2023, 8:36 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 17 2023, 8:36 AM

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

aborrero added a parent task: T307357: Move cloud vps ns-recursor IPs to host/row-independent addressing.Jul 17 2023, 10:34 AM
aborrero edited projects, added cloud-services-team, User-aborrero; removed Cloud-Services.
aborrero subscribed.

We renamed the servers, from .wikimedia.org to .codfw.wmnet.

Are these targets configured in puppet?

aborrero added a subscriber: cmooney.Jul 17 2023, 10:41 AM

I just checked real quick, the ports should be available to the prometheus server:

aborrero@cloudservices2004-dev:~ $ sudo iptables-save -c | grep 10.192.16.75
[58321:3499260] -A INPUT -s 10.192.16.75/32 -j ACCEPT
aborrero@cloudservices2004-dev:~ $ sudo ss -putanl | grep -E 8081\|8082 
tcp   LISTEN 0      10            10.192.20.10:8081       0.0.0.0:*    users:(("pdns_server",pid=1391150,fd=7))                                                                                                                                                                                                                                                                                                   
tcp   LISTEN 0      10            10.192.20.10:8082       0.0.0.0:*    users:(("pdns_recursor",pid=3163970,fd=40))

Maybe the firewalling is happening elsewhere in the switches or core routers, CC @cmooney

aborrero added a comment.Jul 17 2023, 10:42 AM

Yes, I just checked tcpdump and is the return traffic not being accepted.

gerritbot added a comment.Jul 17 2023, 10:45 AM

Change 938819 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/homer/public@master] CR: cloud-host: allow return traffic for PDNS servers

https://gerrit.wikimedia.org/r/938819

gerritbot added a project: Patch-For-Review.Jul 17 2023, 10:45 AM
• taavi added a project: Cloud-VPS.Jul 17 2023, 10:46 AM
aborrero moved this task from Backlog to Doing on the User-aborrero board.Jul 17 2023, 10:53 AM
aborrero moved this task from Doing to Radar/observer on the User-aborrero board.Jul 17 2023, 12:17 PM
fgiunchedi added a comment.Jul 17 2023, 12:35 PM
In T341966#9018932, @aborrero wrote:

We renamed the servers, from .wikimedia.org to .codfw.wmnet.

Are these targets configured in puppet?

Yes the are, however hostnames are searched in all domains both codfw.wmnet and wikimedia.org (JFYI, you already found the root cause anyways!) thank you

gerritbot added a comment.Jul 18 2023, 9:46 AM

Change 938819 merged by Arturo Borrero Gonzalez:

[operations/homer/public@master] CR: cloud-host: allow return traffic for PDNS servers

https://gerrit.wikimedia.org/r/938819

aborrero added a comment.Jul 18 2023, 9:55 AM

Patch deployed, should be fixed now.

fgiunchedi closed this task as Resolved.Jul 18 2023, 9:57 AM
fgiunchedi claimed this task.

Can confirm it is fixed! Thank you @aborrero !

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2023, 10:10 AM
lmata moved this task from Inbox to Radar on the Observability-Metrics board.Jul 18 2023, 5:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits