Page MenuHomePhabricator

netbox DNS Automation Workflow checklist for Commissioning and Decommissioning 2020Q1
Closed, ResolvedPublic

Description

  • interface_automation allocate primary IPs enable switch flipped (PRIMARY_IPS_ENABLED). (At the same time of the 2nd cutoff date in [1])
  • all primary IPs imported to Netbox (prior to the 2nd cutoff date in [1])
    • decide policies for frack import
    • decide policies for other outliers (OIT, RIPE Atlas)
  • interface_automation/import puppetdb works with PRIMARY interfaces post-puppet
  • interface_automation/import puppetdb allows manual fact injection for pre-puppet phase reimage [not needed for now but already implemented]
  • decommission cookbook deletes all non-mgmt ips and interfaces [ @Volans ]
  • document workflows
  • decide timelines [cutoff date decided and announced]
  • Solve potentially circular dependency in Ganeti VM creation [the machine must exist in Netbox before the IP may be allocated, so makevm must be altered to perform the setup tasks there]
  • patch wmf-auto-reimage to run the puppetdb import script for a single host after the first puppet run, how to handle failed scenarios
  • migrate all zonefiles to the automatically generated ones from Netbox (some zonefile might be deemed to not be migrated)

    [1] https://wikitech.wikimedia.org/wiki/DNS/Netbox#Cutoff_dates

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+5 -7
operations/cookbooksmaster+0 -26
operations/software/netbox-extrasmaster+0 -17
operations/dnsmaster+88 -406
operations/cookbooksmaster+2 -2
operations/software/netbox-extrasmaster+1 -1
operations/dnsmaster+64 -2 K
operations/cookbooksmaster+2 -2
operations/software/netbox-extrasmaster+1 -1
operations/dnsmaster+60 -276
operations/cookbooksmaster+2 -2
operations/software/netbox-extrasmaster+1 -1
operations/dnsmaster+61 -222
operations/software/netbox-extrasmaster+2 -0
operations/software/netbox-extrasmaster+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+10 -14
operations/puppetproduction+85 -5
operations/software/netbox-extrasmaster+43 -9
operations/puppetproduction+1 -0
operations/puppetproduction+24 -0
operations/software/netbox-extrasmaster+0 -0
operations/dnsmaster+1 -0
operations/software/netbox-extrasmaster+1 -2
operations/cookbooksmaster+6 -6
operations/dnsmaster+50 -134
operations/dnsmaster+6 -73
operations/cookbooksmaster+5 -1
operations/cookbooksmaster+13 -0
operations/cookbooksmaster+116 -18
operations/software/netbox-extrasmaster+41 -18
operations/cookbooksmaster+18 -3
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 623545 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.ganeti.makevm: adapt to Netbox DNS automation

https://gerrit.wikimedia.org/r/623545

Change 624087 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: allocate IPs, add Cassandra support

https://gerrit.wikimedia.org/r/624087

Change 624087 merged by Volans:
[operations/software/netbox-extras@master] scripts: allocate IPs, add Cassandra support

https://gerrit.wikimedia.org/r/624087

Change 626738 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.hosts.decommission: add Netbox DNS automation

https://gerrit.wikimedia.org/r/626738

Change 623545 merged by jenkins-bot:
[operations/cookbooks@master] sre.ganeti.makevm: adapt to Netbox DNS automation

https://gerrit.wikimedia.org/r/623545

Change 626738 merged by jenkins-bot:
[operations/cookbooks@master] sre.hosts.decommission: add Netbox DNS automation

https://gerrit.wikimedia.org/r/626738

Change 627605 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] Migrate ulsfo records to automated DNS.

https://gerrit.wikimedia.org/r/627605

Change 628046 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Migrate ulsfo public records to automated DNS

https://gerrit.wikimedia.org/r/628046

Change 628144 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: allow to run in DRY-RUN mode

https://gerrit.wikimedia.org/r/628144

Change 628144 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: allow to run in DRY-RUN mode

https://gerrit.wikimedia.org/r/628144

Volans updated the task description. (Show Details)

Change 629055 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] depool ulsfo to migrate its DNS records to Netbox

https://gerrit.wikimedia.org/r/629055

Mentioned in SAL (#wikimedia-operations) [2020-09-22T07:34:02Z] <volans> depooling ulsfo to merge DNS migration to Netbox zonefiles - T258729

Change 629055 merged by Volans:
[operations/dns@master] depool ulsfo to migrate its DNS records to Netbox

https://gerrit.wikimedia.org/r/629055

Mentioned in SAL (#wikimedia-operations) [2020-09-22T07:57:20Z] <volans> migrating ulsfo private DNS records to the Netbox-generated ones - T258729

Change 627605 merged by Volans:
[operations/dns@master] Migrate ulsfo private records to automated DNS

https://gerrit.wikimedia.org/r/627605

Mentioned in SAL (#wikimedia-operations) [2020-09-22T08:22:24Z] <volans> migrating ulsfo public DNS records to the Netbox-generated ones - T258729

Change 628046 merged by Volans:
[operations/dns@master] Migrate ulsfo public records to automated DNS

https://gerrit.wikimedia.org/r/628046

Change 629063 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] multiple: mark ulsfo as migrated to Netbox

https://gerrit.wikimedia.org/r/629063

Change 629064 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: mark ulsfo as migrated to Netbox

https://gerrit.wikimedia.org/r/629064

Change 629063 merged by jenkins-bot:
[operations/cookbooks@master] multiple: mark ulsfo as migrated to Netbox

https://gerrit.wikimedia.org/r/629063

Change 629064 merged by Volans:
[operations/software/netbox-extras@master] scripts: mark ulsfo as migrated to Netbox

https://gerrit.wikimedia.org/r/629064

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:30:52Z] <volans> repooling ulsfo after merging DNS migration to Netbox zonefiles - T258729

Change 629259 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: make the generation script executable

https://gerrit.wikimedia.org/r/629259

Change 629259 merged by Volans:
[operations/software/netbox-extras@master] dns: make the generation script executable

https://gerrit.wikimedia.org/r/629259

Change 629321 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: add check for uncommitted DNS changes

https://gerrit.wikimedia.org/r/629321

Change 629321 merged by Volans:
[operations/puppet@production] netbox: add check for uncommitted DNS changes

https://gerrit.wikimedia.org/r/629321

Change 629358 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: set timeout for nrpe check

https://gerrit.wikimedia.org/r/629358

Change 629358 merged by Volans:
[operations/puppet@production] netbox: set timeout for nrpe check

https://gerrit.wikimedia.org/r/629358

Change 629432 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: convert check Icinga to read/save state

https://gerrit.wikimedia.org/r/629432

Change 629440 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: convert Icinga check in timer

https://gerrit.wikimedia.org/r/629440

Change 629432 merged by Volans:
[operations/software/netbox-extras@master] dns: convert check Icinga to read/save state

https://gerrit.wikimedia.org/r/629432

Change 629440 merged by Volans:
[operations/puppet@production] netbox: convert Icinga check in timer

https://gerrit.wikimedia.org/r/629440

Change 629606 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: improve check_json_file

https://gerrit.wikimedia.org/r/629606

Change 629606 merged by Volans:
[operations/puppet@production] netbox: improve check_json_file

https://gerrit.wikimedia.org/r/629606

Change 629613 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: move state file to /var/run

https://gerrit.wikimedia.org/r/629613

Change 629613 merged by Volans:
[operations/puppet@production] netbox: move state file to /var/run

https://gerrit.wikimedia.org/r/629613

Change 629614 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: fix check on argument

https://gerrit.wikimedia.org/r/629614

Change 629614 merged by Volans:
[operations/software/netbox-extras@master] dns: fix check on argument

https://gerrit.wikimedia.org/r/629614

Change 629694 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: exit with 0 if no changes and --icinga-check

https://gerrit.wikimedia.org/r/629694

Change 629694 merged by Volans:
[operations/software/netbox-extras@master] dns: exit with 0 if no changes and --icinga-check

https://gerrit.wikimedia.org/r/629694

Change 630644 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] Migrate EQSIN to Netbox Automation

https://gerrit.wikimedia.org/r/630644

Change 630647 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] Migrate ESAMS to Netbox Automation

https://gerrit.wikimedia.org/r/630647

Change 630644 merged by CRusnov:
[operations/dns@master] Migrate EQSIN to Netbox Automation

https://gerrit.wikimedia.org/r/630644

Change 630918 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: dns, mark eqsin as migrated to Netbox

https://gerrit.wikimedia.org/r/630918

Change 630919 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Set eqsin as migrated to the DNS Netbox automation

https://gerrit.wikimedia.org/r/630919

Change 630918 merged by Volans:
[operations/software/netbox-extras@master] scripts: dns, mark eqsin as migrated to Netbox

https://gerrit.wikimedia.org/r/630918

Change 630919 merged by jenkins-bot:
[operations/cookbooks@master] Set eqsin as migrated to the DNS Netbox automation

https://gerrit.wikimedia.org/r/630919

Change 631388 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: dns, mark esams as migrated to Netbox

https://gerrit.wikimedia.org/r/631388

Change 631389 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Set esams as migrated to the DNS Netbox automation

https://gerrit.wikimedia.org/r/631389

Mentioned in SAL (#wikimedia-operations) [2020-10-01T16:46:34Z] <volans> migrating esams DNS records to the autogenerated ones from Netbox - T258729

Change 630647 merged by CRusnov:
[operations/dns@master] Migrate ESAMS to Netbox Automation

https://gerrit.wikimedia.org/r/630647

Change 631388 merged by Volans:
[operations/software/netbox-extras@master] scripts: dns, mark esams as migrated to Netbox

https://gerrit.wikimedia.org/r/631388

Change 631389 merged by jenkins-bot:
[operations/cookbooks@master] Set esams as migrated to the DNS Netbox automation

https://gerrit.wikimedia.org/r/631389

Change 634302 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] netbox: Move eqiad private to automation

https://gerrit.wikimedia.org/r/634302

Change 634303 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] netbox: Move eqiad public/svc to automation

https://gerrit.wikimedia.org/r/634303

Change 635865 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Mark eqiad as migrated to Netbox in the DNS

https://gerrit.wikimedia.org/r/635865

Change 635867 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: mark eqiad as migrated to Netbox

https://gerrit.wikimedia.org/r/635867

Change 634302 merged by CRusnov:
[operations/dns@master] netbox: Move eqiad private to automation

https://gerrit.wikimedia.org/r/634302

Mentioned in SAL (#wikimedia-operations) [2020-10-22T17:50:23Z] <volans> cumin 'A:dns-rec' 'rec_control wipe-cache eqiad.wmnet$' - T258729

Change 635867 merged by Volans:
[operations/software/netbox-extras@master] dns: mark eqiad as migrated to Netbox

https://gerrit.wikimedia.org/r/635867

Change 635865 merged by jenkins-bot:
[operations/cookbooks@master] Mark eqiad as migrated to Netbox in the DNS

https://gerrit.wikimedia.org/r/635865

Change 634303 merged by CRusnov:
[operations/dns@master] netbox: Move eqiad public to automation

https://gerrit.wikimedia.org/r/634303

Mentioned in SAL (#wikimedia-operations) [2020-10-22T18:12:12Z] <volans> cumin 'A:dns-rec' 'rec_control wipe-cache wikimedia.org$' - T258729

Mentioned in SAL (#wikimedia-operations) [2020-11-24T09:48:30Z] <volans> Migrating codfw private/public primary DNS records to the auto-generated ones from Netbox - T258729

All datacenters have been migrated, the remaining zones, to be decided if they should be migrated or not, are:

0.3.10.in-addr.arpa
1.2.10.in-addr.arpa
2.2.10.in-addr.arpa
27.35.198.in-addr.arpa
anycast.wmnet
svc.codfw.wmnet
svc.eqiad.wmnet
wikimedia.org-global

As for the wikimediacloud.org records those were left manually managed for now and marked as suck in Netbox to not generate their records. See https://netbox.wikimedia.org/ipam/ip-addresses/?q=wikimediacloud.org

Change 643224 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Remove list of DNS migrated sites

https://gerrit.wikimedia.org/r/643224

Change 643227 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: remove list of DNS migrated sites

https://gerrit.wikimedia.org/r/643227

Change 643227 merged by Volans:
[operations/software/netbox-extras@master] scripts: remove list of DNS migrated sites

https://gerrit.wikimedia.org/r/643227

Change 643224 merged by jenkins-bot:
[operations/cookbooks@master] Remove list of DNS migrated sites

https://gerrit.wikimedia.org/r/643224

27.35.198.in-addr.arpa
wikimedia.org-global

These two are just the fwd+rev for nsa.wikimedia.org. This should be handled the same way as ns[012] are currently, it's basically part of the same set (just not in full production use yet). We could separately debate whether we can/should migrate all of these, but maybe that's out of scope for now?

0.3.10.in-addr.arpa
anycast.wmnet

These are probably migrate-able, I don't see any good reason why we wouldn't.

27.35.198.in-addr.arpa
wikimedia.org-global

These two are just the fwd+rev for nsa.wikimedia.org. This should be handled the same way as ns[012] are currently, it's basically part of the same set (just not in full production use yet). We could separately debate whether we can/should migrate all of these, but maybe that's out of scope for now?

I've marked it as manually managed on Netbox for now so that it doesn't generate the files, and we can adapt based on what we do for the ns records too.

0.3.10.in-addr.arpa
anycast.wmnet

These are probably migrate-able, I don't see any good reason why we wouldn't.

Great, sending the patch shortly.

Change 643321 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] anycast: migrate to the Netbox generated zonefiles

https://gerrit.wikimedia.org/r/643321

The wikimediacloud.org records discussion has been moved to T268621

Change 643321 merged by Volans:
[operations/dns@master] anycast: migrate to the Netbox generated zonefiles

https://gerrit.wikimedia.org/r/643321

Mentioned in SAL (#wikimedia-operations) [2020-11-24T18:56:02Z] <volans> migrating anycast zonefile to the Netbox-generated ones - T258729

The svc ones will be treated in a separate task too, I'm resolving this tracking one as all zones that were supposed to be migrated were migrated.