Page MenuHomePhabricator

tools-redis-1004.tools.eqiad1.wikimedia.cloud resolves to multiple IPs
Closed, ResolvedPublic

Description

$ host tools-redis-1004.tools.eqiad1.wikimedia.cloud
tools-redis-1004.tools.eqiad1.wikimedia.cloud has address 172.16.0.250
tools-redis-1004.tools.eqiad1.wikimedia.cloud has address 172.16.1.169
tools-redis-1004.tools.eqiad1.wikimedia.cloud has address 172.16.1.51
$ host 172.16.0.250
250.0.16.172.in-addr.arpa domain name pointer utrs-playground.utrs.eqiad1.wikimedia.cloud.
250.0.16.172.in-addr.arpa domain name pointer tools-redis-1004.tools.eqiad1.wikimedia.cloud.
$ host 172.16.1.169
169.1.16.172.in-addr.arpa domain name pointer tools-redis-1004.tools.eqiad1.wikimedia.cloud.
169.1.16.172.in-addr.arpa domain name pointer sdcquery01.wikidata-query.eqiad1.wikimedia.cloud.
$ host 172.16.1.51
51.1.16.172.in-addr.arpa domain name pointer tools-redis-1004.tools.eqiad1.wikimedia.cloud.

Event Timeline

bd808 triaged this task as Medium priority.Sep 9 2020, 12:00 AM

Following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#.wmflabs_and_.wikimedia.cloud

$ ssh cloudcontrol1003.wikimedia.org
$ sudo su -
$ source /root/novaenv.sh
$ OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1004.tools.eqiad1.wikimedia.cloud. --record 172.16.1.51 --edit-managed
url: /zones/67603ef4-3d64-40d6-90d3-5b7776a99034/recordsets/01782a28-3e5e-46a4-a66a-44467bf4d21a
kwargs: {'data': '{"records": ["172.16.1.51"]}'}
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| action      | UPDATE                                         |
| created_at  | 2020-04-03T18:52:05.000000                     |
| description | None                                           |
| id          | 01782a28-3e5e-46a4-a66a-44467bf4d21a           |
| name        | tools-redis-1004.tools.eqiad1.wikimedia.cloud. |
| project_id  | cloudinfra                                     |
| records     | 172.16.1.51                                    |
| status      | PENDING                                        |
| ttl         | None                                           |
| type        | A                                              |
| updated_at  | 2020-09-08T23:58:57.000000                     |
| version     | 2                                              |
| zone_id     | 67603ef4-3d64-40d6-90d3-5b7776a99034           |
| zone_name   | eqiad1.wikimedia.cloud.                        |
+-------------+------------------------------------------------+
$ host tools-redis-1004.tools.eqiad1.wikimedia.cloud
tools-redis-1004.tools.eqiad1.wikimedia.cloud has address 172.16.1.51

Mentioned in SAL (#wikimedia-cloud) [2020-09-09T00:05:29Z] <bd808> Running wmcs-novastats-dnsleaks (T262359)

The only other multiple address record that wmcs-novastats-dnsleaks found was for canary1036-01.eqiad.wmflabs

$ wmcs-novastats-dnsleaks
WARNING:root:You are loading the python2 version of this library.  Time to upgrade your script to python3!
A record for canary1036-01.eqiad.wmflabs. has multiple IPs: ['172.16.2.51', '172.16.2.6']
This needs cleanup but that isn't implemented and almost never happens.
$ openstack recordset set eqiad.wmflabs. canary1036-01.eqiad.wmflabs. --record 172.16.2.6 --edit-managed --sudo-project-id noauth-project
url: /zones/114f1333-c2c1-44d3-beb4-ebed1a91742b/recordsets/14b3bac5-2a0c-48fb-841a-e4dbc4ce147e
kwargs: {'data': '{"records": ["172.16.2.6"]}'}
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| action      | UPDATE                               |
| created_at  | 2020-08-01T00:36:15.000000           |
| description | None                                 |
| id          | 14b3bac5-2a0c-48fb-841a-e4dbc4ce147e |
| name        | canary1036-01.eqiad.wmflabs.         |
| project_id  | noauth-project                       |
| records     | 172.16.2.6                           |
| status      | PENDING                              |
| ttl         | None                                 |
| type        | A                                    |
| updated_at  | 2020-09-09T00:21:35.000000           |
| version     | 2                                    |
| zone_id     | 114f1333-c2c1-44d3-beb4-ebed1a91742b |
| zone_name   | eqiad.wmflabs.                       |
+-------------+--------------------------------------+

@Andrew before I close this out, do you have any hunches about what might have caused the bad record for tools-redis-1004.tools.eqiad1.wikimedia.cloud in the first place?

@Andrew before I close this out, do you have any hunches about what might have caused the bad record for tools-redis-1004.tools.eqiad1.wikimedia.cloud in the first place?

I mostly don't. Until we adopt synchronous DNS management there will probably always be a few leaks like this as services start and stop.