Page MenuHomePhabricator

Fix recdns config on various hardware devices
Closed, ResolvedPublic

Description

There are two legacy recdns IPs which are still supported (for now) and still in active use by a few devices: 208.80.153.254 and 208.80.154.254. These should both be replaced in all config with with 10.3.0.1 instead. Most of the migration was done months ago, but some hardware devices remain on the old config. A recent 1-week packet capture survey turned up two clients:

  • scs-a1-codfw.mgmt.codfw.wmnet
  • ps1-b5-eqiad.mgmt.eqiad.wmnet

Additionally, SYN packets were found from a "WMCS eqiad/codfw + drmrs + esams OOB" client on both DNS servers:

IP 185.15.56.1.38304 > 208.80.153.254.53: Flags [S], seq 2423093459, win 1024, options [mss 1460], length 0 (WMCS eqiad/codfw + drmrs + esams OOB)
IP 185.15.56.1.52830 > 208.80.154.254.53: Flags [S], seq 3220011912, win 1024, options [mss 1460], length 0 (WMCS eqiad/codfw + drmrs + esams OOB)

Related Objects

Event Timeline

Dzahn triaged this task as Medium priority.Jun 4 2020, 9:19 AM

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

I've updated the list based on a recent week-long tcpdump on both DNS servers.

@Papaul, could you update scs-a1-codfw.mgmt.codfw.wmnet to point to the newer 10.3.0.1 DNS, please?

@Jclark-ctr, could you update ps1-b5-eqiad.mgmt.eqiad.wmnet to point to the newer 10.3.0.1 DNS, please?

@BCornwall I took care of both of them :)

185.15.56.1 is the generic NAT IP for the WMCS realm.

@aborrero, for context, 208.80.153.254 is our old recursive DNS IP and it's being decommissioned (replaced with the internal 10.3.0.1 IP). As WMCS have its own recursive DNS I think those flows are bogus, but we need to make sure it won't break anything when we decom' them. Is it possible to know which VM try to reach them and why?

In a quick search using cumin I didn't find anything relevant:

aborrero@cloud-cumin-03:~$ sudo cumin --force -x '*' "grep 208.80.154.254 /etc/resolv.conf"
[.. nothing relevant ..]

I need to investigate more.

NAT logs or tcpdump on the device doing NAT should help pinpoint the host(s).

It is diffscan02.automation-framework.eqiad1.wikimedia.cloud.

There are 1k connections like this:

tcp      6 59 SYN_SENT src=172.16.3.44 dst=208.80.154.254 sport=59626 dport=35106 [UNREPLIED] src=208.80.154.254 dst=185.15.56.1 sport=35106 dport=59626 mark=0 use=1

With 172.16.3.44 being that VM.

Nice! that makes sens as it scans all our IPs.

@BCornwall I think everything is completed here!

Thanks for handling that, @ayounsi!