Page MenuHomePhabricator
Create Task
Maniphest T221183

Rename and re-assign cloud dns servers
Closed, ResolvedPublic
Actions

Description

We're going to switch off labservices1001 and labservices1002 pretty soon. This is an opportunity to create new 'cloud-' name server addresses and move all the load onto cloudservices1003 and cloudservices1004.

The actual DNS content is synced between all four servers, so we can safely switch things over as DNS updates.

before

labservices1001

208.80.155.117: labservices1001.wikimedia.org, labs-ns0.wikimedia.org
208.80.155.118: labs-recursor0.wikimedia.org

labservices1002

208.80.154.12: labservices1002.wikimedia.org, labs-ns1.wikimedia.org
208.80.154.20: labs-recursor1.wikimedia.org

cloudservices1003

208.80.154.135: cloudservices1003.wikimedia.org, labs-ns2.wikimedia.org
208.80.154.143: labs-recursor2.wikimedia.org

cloudservices1004

208.80.154.11: cloudservices1004.wikimedia.org, labs-ns3.wikimedia.org
208.80.154.24: labs-recursor3.wikimedia.org

cloud vm resolv.conf:

nameserver 208.80.155.118
nameserver 208.80.154.20

after

labservices1001

208.80.155.117: labservices1001.wikimedia.org

labservices1002

208.80.154.12: labservices1002.wikimedia.org

cloudservices1003

208.80.154.135: cloudservices1003.wikimedia.org, cloud-ns0.wikimedia.org
208.80.154.143: cloud-recursor0.wikimedia.org

cloudservices1004

208.80.154.11: cloudservices1004.wikimedia.org, cloud-ns1.wikimedia.org
208.80.154.24: cloud-recursor1.wikimedia.org

cloud vm resolv.conf:

nameserver 208.80.154.143
nameserver 208.80.154.24

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add ptr records for cloud-recursor{0,1}operations/dnsmaster+2 -0
Remove labs-ns and labs-recursor namesoperations/dnsmaster+0 -12
Temporary hack: turn off designate on labservices1001operations/puppetproduction+3 -3
Remove old labs 'main' region in-addr.arpa delegationoperations/dnsmaster+0 -135
Remove delegation of 208.80.155.128-255operations/dnsmaster+0 -135
cloud dns: move primary services to cloud-ns0 and cloud-ns1operations/puppetproduction+48 -113
cloud VMS: move primary resolvers to cloud-recursor0/1operations/puppetproduction+2 -2
cloud: add new cloud-ns and cloud-recursor names to cloudservices hostsoperations/dnsmaster+6 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Apr 17 2019, 3:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 17 2019, 3:41 AM
Andrew triaged this task as Medium priority.Apr 17 2019, 3:41 AM
Andrew added a project: cloud-services-team (Kanban).
gerritbot added a comment.Apr 17 2019, 3:46 AM

Change 504490 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] cloud: add new cloud-ns and cloud-recursor names to cloudservices hosts

https://gerrit.wikimedia.org/r/504490

gerritbot added a project: Patch-For-Review.Apr 17 2019, 3:46 AM
gerritbot added a comment.Apr 17 2019, 2:36 PM

Change 504490 merged by Andrew Bogott:
[operations/dns@master] cloud: add new cloud-ns and cloud-recursor names to cloudservices hosts

https://gerrit.wikimedia.org/r/504490

gerritbot added a comment.Apr 17 2019, 2:56 PM

Change 504572 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud dns: move primary services to cloud-ns0 and cloud-ns1

https://gerrit.wikimedia.org/r/504572

gerritbot added a comment.Apr 17 2019, 3:15 PM

Change 504580 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud VMS: move primary resolvers to cloud-recursor0/1

https://gerrit.wikimedia.org/r/504580

gerritbot added a comment.Apr 18 2019, 2:58 PM

Change 504580 merged by Andrew Bogott:
[operations/puppet@production] cloud VMS: move primary resolvers to cloud-recursor0/1

https://gerrit.wikimedia.org/r/504580

Andrew added a comment.EditedApr 20 2019, 2:32 PM

Most VMs are now using the new recursors. 120+ had broken puppet and so didn't pick up the change... I fixed puppet myself on most of them but there are still 30-some needing fixes, for which I've opened a bunch of bugs. Worst case I guess we can just sed the new IP into each of the broken ones.

Remaining steps are...

  • merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/504572/
  • update markmonitor T221240 -- PENDING
  • Apply the new pool configs on cloudservices1003 and 1004 so they don't try to sync with labservices1001/1002 anymore
  • Update SOA records to use the new cloud-ns0 name (this will involve mucking in the designate database)
  • Shut down services on labservices1001/1002, test, rebuild them with role::spare, etc.
Andrew mentioned this in T212772: Track remaining trusty servers in production.Apr 20 2019, 2:49 PM
gerritbot added a comment.Apr 21 2019, 4:17 PM

Change 504572 merged by Andrew Bogott:
[operations/puppet@production] cloud dns: move primary services to cloud-ns0 and cloud-ns1

https://gerrit.wikimedia.org/r/504572

Andrew mentioned this in T221654: Puppet broken on VMs in deployment-prep.Apr 23 2019, 3:42 PM
ayounsi closed subtask T221531: Update RIPE about changes in WMCS auth servers as Resolved.Apr 23 2019, 8:58 PM
Andrew added a comment.EditedApr 23 2019, 9:05 PM

With ripe's 2-day ttl, we should leave the old labs-ns0/1 servers up until 48 hours from this timestamp. Markmonitor will be well-since updated by then.

gerritbot added a comment.Apr 25 2019, 2:31 PM

Change 506436 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] Remove delegation of 208.80.155.128-255

https://gerrit.wikimedia.org/r/506436

Andrew closed subtask T221240: Update markmonitor with new dns info for .wmflabs.org as Resolved.Apr 25 2019, 2:31 PM
gerritbot added a comment.Apr 25 2019, 2:39 PM

Change 506436 abandoned by Andrew Bogott:
Remove delegation of 208.80.155.128-255

Reason:
dropping in favor of https://gerrit.wikimedia.org/r/#/c/operations/dns/ /505478/

https://gerrit.wikimedia.org/r/506436

gerritbot added a comment.Apr 25 2019, 2:40 PM

Change 505478 had a related patch set uploaded (by Andrew Bogott; owner: Alex Monk):
[operations/dns@master] Remove old labs 'main' region in-addr.arpa delegation

https://gerrit.wikimedia.org/r/505478

gerritbot added a comment.Apr 25 2019, 2:47 PM

Change 505478 merged by Andrew Bogott:
[operations/dns@master] Remove old labs 'main' region in-addr.arpa delegation

https://gerrit.wikimedia.org/r/505478

gerritbot added a comment.Apr 26 2019, 2:44 PM

Change 506686 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Temporary hack: turn off designate on labservices1001

https://gerrit.wikimedia.org/r/506686

gerritbot added a comment.Apr 26 2019, 2:45 PM

Change 506686 merged by Andrew Bogott:
[operations/puppet@production] Temporary hack: turn off designate on labservices1001

https://gerrit.wikimedia.org/r/506686

Andrew mentioned this in T223370: CloudVPS DNS: weird behavior in integration-puppetmaster01.May 15 2019, 2:40 PM
gerritbot added a comment.May 15 2019, 2:56 PM

Change 510546 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] Remove labs-ns and labs-recursor names

https://gerrit.wikimedia.org/r/510546

gerritbot added a comment.May 15 2019, 5:27 PM

Change 510546 merged by Andrew Bogott:
[operations/dns@master] Remove labs-ns and labs-recursor names

https://gerrit.wikimedia.org/r/510546

gerritbot added a comment.May 15 2019, 6:05 PM

Change 510583 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] Add ptr records for cloud-recursor{0,1}

https://gerrit.wikimedia.org/r/510583

gerritbot added a comment.May 15 2019, 6:06 PM

Change 510583 merged by Andrew Bogott:
[operations/dns@master] Add ptr records for cloud-recursor{0,1}

https://gerrit.wikimedia.org/r/510583

Andrew claimed this task.May 15 2019, 6:11 PM
Andrew closed this task as Resolved.May 15 2019, 6:16 PM
• Bstorm closed subtask T221721: Puppet broken on several vms in toolsbeta as Resolved.Jun 17 2019, 3:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits