$ host tools-redis-1004.tools.eqiad1.wikimedia.cloud tools-redis-1004.tools.eqiad1.wikimedia.cloud has address 172.16.0.250 tools-redis-1004.tools.eqiad1.wikimedia.cloud has address 172.16.1.169 tools-redis-1004.tools.eqiad1.wikimedia.cloud has address 172.16.1.51 $ host 172.16.0.250 250.0.16.172.in-addr.arpa domain name pointer utrs-playground.utrs.eqiad1.wikimedia.cloud. 250.0.16.172.in-addr.arpa domain name pointer tools-redis-1004.tools.eqiad1.wikimedia.cloud. $ host 172.16.1.169 169.1.16.172.in-addr.arpa domain name pointer tools-redis-1004.tools.eqiad1.wikimedia.cloud. 169.1.16.172.in-addr.arpa domain name pointer sdcquery01.wikidata-query.eqiad1.wikimedia.cloud. $ host 172.16.1.51 51.1.16.172.in-addr.arpa domain name pointer tools-redis-1004.tools.eqiad1.wikimedia.cloud.
Description
Description
Event Timeline
Comment Actions
$ ssh cloudcontrol1003.wikimedia.org $ sudo su - $ source /root/novaenv.sh $ OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1004.tools.eqiad1.wikimedia.cloud. --record 172.16.1.51 --edit-managed url: /zones/67603ef4-3d64-40d6-90d3-5b7776a99034/recordsets/01782a28-3e5e-46a4-a66a-44467bf4d21a kwargs: {'data': '{"records": ["172.16.1.51"]}'} +-------------+------------------------------------------------+ | Field | Value | +-------------+------------------------------------------------+ | action | UPDATE | | created_at | 2020-04-03T18:52:05.000000 | | description | None | | id | 01782a28-3e5e-46a4-a66a-44467bf4d21a | | name | tools-redis-1004.tools.eqiad1.wikimedia.cloud. | | project_id | cloudinfra | | records | 172.16.1.51 | | status | PENDING | | ttl | None | | type | A | | updated_at | 2020-09-08T23:58:57.000000 | | version | 2 | | zone_id | 67603ef4-3d64-40d6-90d3-5b7776a99034 | | zone_name | eqiad1.wikimedia.cloud. | +-------------+------------------------------------------------+
$ host tools-redis-1004.tools.eqiad1.wikimedia.cloud tools-redis-1004.tools.eqiad1.wikimedia.cloud has address 172.16.1.51
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-09-09T00:05:29Z] <bd808> Running wmcs-novastats-dnsleaks (T262359)
Comment Actions
The only other multiple address record that wmcs-novastats-dnsleaks found was for canary1036-01.eqiad.wmflabs
$ wmcs-novastats-dnsleaks WARNING:root:You are loading the python2 version of this library. Time to upgrade your script to python3! A record for canary1036-01.eqiad.wmflabs. has multiple IPs: ['172.16.2.51', '172.16.2.6'] This needs cleanup but that isn't implemented and almost never happens. $ openstack recordset set eqiad.wmflabs. canary1036-01.eqiad.wmflabs. --record 172.16.2.6 --edit-managed --sudo-project-id noauth-project url: /zones/114f1333-c2c1-44d3-beb4-ebed1a91742b/recordsets/14b3bac5-2a0c-48fb-841a-e4dbc4ce147e kwargs: {'data': '{"records": ["172.16.2.6"]}'} +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | action | UPDATE | | created_at | 2020-08-01T00:36:15.000000 | | description | None | | id | 14b3bac5-2a0c-48fb-841a-e4dbc4ce147e | | name | canary1036-01.eqiad.wmflabs. | | project_id | noauth-project | | records | 172.16.2.6 | | status | PENDING | | ttl | None | | type | A | | updated_at | 2020-09-09T00:21:35.000000 | | version | 2 | | zone_id | 114f1333-c2c1-44d3-beb4-ebed1a91742b | | zone_name | eqiad.wmflabs. | +-------------+--------------------------------------+
Comment Actions
@Andrew before I close this out, do you have any hunches about what might have caused the bad record for tools-redis-1004.tools.eqiad1.wikimedia.cloud in the first place?
Comment Actions
I mostly don't. Until we adopt synchronous DNS management there will probably always be a few leaks like this as services start and stop.