Page MenuHomePhabricator
Create Task
Maniphest T246551

DNS cleanup for deleted instances in codfw1dev is broken
Closed, ResolvedPublic
Actions

Description

labtestkrenair@cloudinfra-db-01:~$ host 172.16.128.14
14.128.16.172.in-addr.arpa domain name pointer jeh-test6.admin.codfw1dev.cloud.
14.128.16.172.in-addr.arpa domain name pointer newton-18.devtest.codfw1dev.cloud.
14.128.16.172.in-addr.arpa domain name pointer jeh-test2.admin.codfw1dev.cloud.
14.128.16.172.in-addr.arpa domain name pointer bastion-codfw1dev-01.bastioninfra-codfw1dev.codfw1dev.cloud.
labtestkrenair@cloudinfra-db-01:~$ host 172.16.128.19
19.128.16.172.in-addr.arpa domain name pointer bastion-codfw1dev-02.bastioninfra-codfw1dev.codfw1dev.wikimedia.cloud.
19.128.16.172.in-addr.arpa domain name pointer testinstance-16.devtest.codfw1dev.cloud.
19.128.16.172.in-addr.arpa domain name pointer sinktest-106.andrewtestproject.codfw1dev.cloud.
19.128.16.172.in-addr.arpa domain name pointer newton-06.devtest.codfw1dev.cloud.
19.128.16.172.in-addr.arpa domain name pointer newton-11.devtest.codfw1dev.cloud.
19.128.16.172.in-addr.arpa domain name pointer sinktest-601.andrewtestproject.codfw1dev.cloud.

The incorrect A records are still there too:

labtestkrenair@cloudinfra-db-01:~$ host jeh-test6.admin.codfw1dev.cloud
jeh-test6.admin.codfw1dev.cloud has address 172.16.128.14
labtestkrenair@cloudinfra-db-01:~$ host newton-18.devtest.codfw1dev.cloud
newton-18.devtest.codfw1dev.cloud has address 172.16.128.14
labtestkrenair@cloudinfra-db-01:~$ host jeh-test2.admin.codfw1dev.cloud
jeh-test2.admin.codfw1dev.cloud has address 172.16.128.14
labtestkrenair@cloudinfra-db-01:~$ host testinstance-16.devtest.codfw1dev.cloud
testinstance-16.devtest.codfw1dev.cloud has address 172.16.128.19
labtestkrenair@cloudinfra-db-01:~$ host sinktest-106.andrewtestproject.codfw1dev.cloud
sinktest-106.andrewtestproject.codfw1dev.cloud has address 172.16.128.19
labtestkrenair@cloudinfra-db-01:~$ host newton-06.devtest.codfw1dev.cloud
newton-06.devtest.codfw1dev.cloud has address 172.16.128.19
newton-06.devtest.codfw1dev.cloud has address 172.16.128.20
labtestkrenair@cloudinfra-db-01:~$ host newton-11.devtest.codfw1dev.cloud
newton-11.devtest.codfw1dev.cloud has address 172.16.128.19
labtestkrenair@cloudinfra-db-01:~$ host sinktest-601.andrewtestproject.codfw1dev.cloud
sinktest-601.andrewtestproject.codfw1dev.cloud has address 172.16.128.19

Details

SubjectRepoBranchLines +/-
wmcs-novastats-dnsleaks: make safe to run in codfw1devoperations/puppetproduction+26 -15
wmcs-novastats-dnsleaks: run through 2to3 and blackoperations/puppetproduction+90 -60
Customize query in gerrit

Related Objects

Event Timeline

Krenair created this task.Feb 29 2020, 9:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 29 2020, 9:50 PM
bd808 edited projects, added cloud-services-team (Kanban), Cloud-VPS; removed cloud-services-team.Feb 29 2020, 9:56 PM
Andrew claimed this task.Mar 2 2020, 8:56 PM
gerritbot added a comment.Mar 2 2020, 9:53 PM

Change 576154 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs-novastats-dnsleaks: make safe to run in codfw1dev

https://gerrit.wikimedia.org/r/576154

gerritbot added a project: Patch-For-Review.Mar 2 2020, 9:53 PM
gerritbot added a comment.Mar 2 2020, 10:01 PM

Change 576155 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs-novastats-dnsleaks: make safe to run in codfw1dev

https://gerrit.wikimedia.org/r/576155

Andrew added a comment.Mar 2 2020, 10:05 PM

I ran a big cleanup on codfw1dev:

root@bastion-codfw1dev-02:~# host 172.16.128.14
14.128.16.172.in-addr.arpa domain name pointer bastion-codfw1dev-01.bastioninfra-codfw1dev.codfw1dev.cloud.
root@bastion-codfw1dev-02:~# host jeh-test6.admin.codfw1dev.cloud
Host jeh-test6.admin.codfw1dev.cloud not found: 3(NXDOMAIN)
root@bastion-codfw1dev-02:~# host newton-18.devtest.codfw1dev.cloud
Host newton-18.devtest.codfw1dev.cloud not found: 3(NXDOMAIN)

I also verified that a new vm creates and then deletes its records properly. Please re-open if you find things I missed or see this happening again.

gerritbot added a comment.Mar 2 2020, 10:17 PM

Change 576154 merged by Andrew Bogott:
[operations/puppet@production] wmcs-novastats-dnsleaks: run through 2to3 and black

https://gerrit.wikimedia.org/r/576154

gerritbot added a comment.Mar 3 2020, 3:33 AM

Change 576155 merged by Andrew Bogott:
[operations/puppet@production] wmcs-novastats-dnsleaks: make safe to run in codfw1dev

https://gerrit.wikimedia.org/r/576155

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2020, 4:10 AM
Krenair closed this task as Resolved.Mar 6 2020, 10:23 PM
Krenair mentioned this in T247134: Fix all codfw1dev.cloud references.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL