- interface_automation allocate primary IPs enable switch flipped (PRIMARY_IPS_ENABLED). (At the same time of the 2nd cutoff date in [1])
- all primary IPs imported to Netbox (prior to the 2nd cutoff date in [1])
- decide policies for frack import
- decide policies for other outliers (OIT, RIPE Atlas)
- interface_automation/import puppetdb works with PRIMARY interfaces post-puppet
- interface_automation/import puppetdb allows manual fact injection for pre-puppet phase reimage [not needed for now but already implemented]
- decommission cookbook deletes all non-mgmt ips and interfaces [ @Volans ]
- document workflows
- decide timelines [cutoff date decided and announced]
- Solve potentially circular dependency in Ganeti VM creation [the machine must exist in Netbox before the IP may be allocated, so makevm must be altered to perform the setup tasks there]
- patch wmf-auto-reimage to run the puppetdb import script for a single host after the first puppet run, how to handle failed scenarios
- migrate all zonefiles to the automatically generated ones from Netbox (some zonefile might be deemed to not be migrated)
[1] https://wikitech.wikimedia.org/wiki/DNS/Netbox#Cutoff_dates
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | • crusnov | T258729 netbox DNS Automation Workflow checklist for Commissioning and Decommissioning 2020Q1 | |||
Resolved | • crusnov | T244153 Import IP addresses, interfaces and DNS names into Netbox for Primary Interfaces | |||
Resolved | Volans | T264273 DNS: per prefix zone-file limitation |
Event Timeline
Change 623545 merged by jenkins-bot:
[operations/cookbooks@master] sre.ganeti.makevm: adapt to Netbox DNS automation
Change 626738 merged by jenkins-bot:
[operations/cookbooks@master] sre.hosts.decommission: add Netbox DNS automation
Change 627605 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] Migrate ulsfo records to automated DNS.
Change 628046 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Migrate ulsfo public records to automated DNS
Change 628144 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: allow to run in DRY-RUN mode
Change 628144 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: allow to run in DRY-RUN mode
Change 629055 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] depool ulsfo to migrate its DNS records to Netbox
Mentioned in SAL (#wikimedia-operations) [2020-09-22T07:34:02Z] <volans> depooling ulsfo to merge DNS migration to Netbox zonefiles - T258729
Change 629055 merged by Volans:
[operations/dns@master] depool ulsfo to migrate its DNS records to Netbox
Mentioned in SAL (#wikimedia-operations) [2020-09-22T07:57:20Z] <volans> migrating ulsfo private DNS records to the Netbox-generated ones - T258729
Change 627605 merged by Volans:
[operations/dns@master] Migrate ulsfo private records to automated DNS
Mentioned in SAL (#wikimedia-operations) [2020-09-22T08:22:24Z] <volans> migrating ulsfo public DNS records to the Netbox-generated ones - T258729
Change 628046 merged by Volans:
[operations/dns@master] Migrate ulsfo public records to automated DNS
Change 629063 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] multiple: mark ulsfo as migrated to Netbox
Change 629064 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: mark ulsfo as migrated to Netbox
Change 629063 merged by jenkins-bot:
[operations/cookbooks@master] multiple: mark ulsfo as migrated to Netbox
Change 629064 merged by Volans:
[operations/software/netbox-extras@master] scripts: mark ulsfo as migrated to Netbox
Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:30:52Z] <volans> repooling ulsfo after merging DNS migration to Netbox zonefiles - T258729
Change 629259 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: make the generation script executable
Change 629259 merged by Volans:
[operations/software/netbox-extras@master] dns: make the generation script executable
Change 629321 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: add check for uncommitted DNS changes
Change 629321 merged by Volans:
[operations/puppet@production] netbox: add check for uncommitted DNS changes
Change 629358 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: set timeout for nrpe check
Change 629358 merged by Volans:
[operations/puppet@production] netbox: set timeout for nrpe check
Change 629432 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: convert check Icinga to read/save state
Change 629440 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: convert Icinga check in timer
Change 629432 merged by Volans:
[operations/software/netbox-extras@master] dns: convert check Icinga to read/save state
Change 629440 merged by Volans:
[operations/puppet@production] netbox: convert Icinga check in timer
Change 629606 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: improve check_json_file
Change 629606 merged by Volans:
[operations/puppet@production] netbox: improve check_json_file
Change 629613 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: move state file to /var/run
Change 629613 merged by Volans:
[operations/puppet@production] netbox: move state file to /var/run
Change 629614 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: fix check on argument
Change 629614 merged by Volans:
[operations/software/netbox-extras@master] dns: fix check on argument
Change 629694 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: exit with 0 if no changes and --icinga-check
Change 629694 merged by Volans:
[operations/software/netbox-extras@master] dns: exit with 0 if no changes and --icinga-check
Change 630644 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] Migrate EQSIN to Netbox Automation
Change 630647 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] Migrate ESAMS to Netbox Automation
Change 630644 merged by CRusnov:
[operations/dns@master] Migrate EQSIN to Netbox Automation
Change 630918 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: dns, mark eqsin as migrated to Netbox
Change 630919 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Set eqsin as migrated to the DNS Netbox automation
Change 630918 merged by Volans:
[operations/software/netbox-extras@master] scripts: dns, mark eqsin as migrated to Netbox
Change 630919 merged by jenkins-bot:
[operations/cookbooks@master] Set eqsin as migrated to the DNS Netbox automation
Change 631388 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: dns, mark esams as migrated to Netbox
Change 631389 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Set esams as migrated to the DNS Netbox automation
Mentioned in SAL (#wikimedia-operations) [2020-10-01T16:46:34Z] <volans> migrating esams DNS records to the autogenerated ones from Netbox - T258729
Change 630647 merged by CRusnov:
[operations/dns@master] Migrate ESAMS to Netbox Automation
Change 631388 merged by Volans:
[operations/software/netbox-extras@master] scripts: dns, mark esams as migrated to Netbox
Change 631389 merged by jenkins-bot:
[operations/cookbooks@master] Set esams as migrated to the DNS Netbox automation
Change 634302 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] netbox: Move eqiad private to automation
Change 634303 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/dns@master] netbox: Move eqiad public/svc to automation
Change 635865 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Mark eqiad as migrated to Netbox in the DNS
Change 635867 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: mark eqiad as migrated to Netbox
Change 634302 merged by CRusnov:
[operations/dns@master] netbox: Move eqiad private to automation
Mentioned in SAL (#wikimedia-operations) [2020-10-22T17:50:23Z] <volans> cumin 'A:dns-rec' 'rec_control wipe-cache eqiad.wmnet$' - T258729
Change 635867 merged by Volans:
[operations/software/netbox-extras@master] dns: mark eqiad as migrated to Netbox
Change 635865 merged by jenkins-bot:
[operations/cookbooks@master] Mark eqiad as migrated to Netbox in the DNS
Change 634303 merged by CRusnov:
[operations/dns@master] netbox: Move eqiad public to automation
Mentioned in SAL (#wikimedia-operations) [2020-10-22T18:12:12Z] <volans> cumin 'A:dns-rec' 'rec_control wipe-cache wikimedia.org$' - T258729
Mentioned in SAL (#wikimedia-operations) [2020-11-24T09:48:30Z] <volans> Migrating codfw private/public primary DNS records to the auto-generated ones from Netbox - T258729
All datacenters have been migrated, the remaining zones, to be decided if they should be migrated or not, are:
0.3.10.in-addr.arpa 1.2.10.in-addr.arpa 2.2.10.in-addr.arpa 27.35.198.in-addr.arpa anycast.wmnet svc.codfw.wmnet svc.eqiad.wmnet wikimedia.org-global
As for the wikimediacloud.org records those were left manually managed for now and marked as suck in Netbox to not generate their records. See https://netbox.wikimedia.org/ipam/ip-addresses/?q=wikimediacloud.org
Change 643224 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Remove list of DNS migrated sites
Change 643227 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: remove list of DNS migrated sites
Change 643227 merged by Volans:
[operations/software/netbox-extras@master] scripts: remove list of DNS migrated sites
Change 643224 merged by jenkins-bot:
[operations/cookbooks@master] Remove list of DNS migrated sites
These two are just the fwd+rev for nsa.wikimedia.org. This should be handled the same way as ns[012] are currently, it's basically part of the same set (just not in full production use yet). We could separately debate whether we can/should migrate all of these, but maybe that's out of scope for now?
0.3.10.in-addr.arpa
anycast.wmnet
These are probably migrate-able, I don't see any good reason why we wouldn't.
I've marked it as manually managed on Netbox for now so that it doesn't generate the files, and we can adapt based on what we do for the ns records too.
0.3.10.in-addr.arpa
anycast.wmnetThese are probably migrate-able, I don't see any good reason why we wouldn't.
Great, sending the patch shortly.
Change 643321 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] anycast: migrate to the Netbox generated zonefiles
Change 643321 merged by Volans:
[operations/dns@master] anycast: migrate to the Netbox generated zonefiles
Mentioned in SAL (#wikimedia-operations) [2020-11-24T18:56:02Z] <volans> migrating anycast zonefile to the Netbox-generated ones - T258729
The svc ones will be treated in a separate task too, I'm resolving this tracking one as all zones that were supposed to be migrated were migrated.