Page MenuHomePhabricator

Automate generation of Management DNS records from Netbox
Closed, ResolvedPublic

Description

Get to the "testing" phase of automated generation:

  • Read Netbox API
  • Dump includable records and reverse records for Management interfaces
  • Test and verify produced records against manually maintained records

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+22 -3 K
operations/dnsmaster+0 -4
operations/software/netbox-extrasmaster+6 -4
operations/software/netbox-extrasmaster+1 -1
operations/dnsmaster+19 -2 K
operations/dnsmaster+2 -100
operations/dnsmaster+2 -74
operations/software/netbox-extrasmaster+1 -1
operations/software/netbox-extrasmaster+292 -72
operations/dnsmaster+7 -132
operations/dnsmaster+6 -112
operations/dnsmaster+6 -113
operations/software/netbox-extrasmaster+57 -24
operations/cookbooksmaster+7 -4
operations/cookbooksmaster+6 -3
operations/dnsmaster+0 -91
operations/dnsmaster+0 -34
operations/dnsmaster+2 -2
operations/cookbooksmaster+5 -1
operations/cookbooksmaster+7 -5
operations/cookbooksmaster+6 -1
operations/cookbooksmaster+3 -4
operations/software/netbox-extrasmaster+1 -1
operations/cookbooksmaster+3 -3
operations/cookbooksmaster+3 -2
operations/cookbooksmaster+3 -1
operations/software/netbox-extrasmaster+2 -4
operations/cookbooksmaster+77 -0
operations/puppetproduction+34 -1
operations/software/netbox-extrasmaster+90 -32
operations/software/netbox-extrasmaster+1 -1
operations/software/netbox-extrasmaster+15 -6
operations/software/netbox-extrasmaster+230 -94
operations/software/netbox-extrasmaster+45 -30
operations/software/netbox-extrasmaster+8 -4
operations/software/netbox-extrasmaster+6 -4
operations/software/netbox-extrasmaster+1 -1
operations/software/netbox-extrasmaster+17 -10
operations/software/netbox-extrasmaster+9 -2
operations/software/netbox-extrasmaster+1 -1
operations/puppetproduction+2 -1
operations/puppetproduction+23 -3
operations/software/netbox-extrasmaster+1 -1
operations/software/netbox-extrasmaster+11 -3
operations/software/netbox-extrasmaster+35 -15
operations/puppetproduction+3 -4
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+162 -5
operations/dnsmaster+1 -1
operations/dnsmaster+1 -0
operations/software/netbox-extrasmaster+295 -0
integration/configmaster+1 -1
operations/puppetproduction+69 -0
operations/software/netbox-deploymaster+355 -0
integration/configmaster+4 -0
operations/puppetproduction+78 -0
operations/puppetproduction+312 -1
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 576985 merged by Volans:
[operations/software/netbox-extras@master] dns: convert Netbox data gathering into a class

https://gerrit.wikimedia.org/r/576985

Change 576986 merged by Volans:
[operations/software/netbox-extras@master] dns: convert records management in classes

https://gerrit.wikimedia.org/r/576986

Change 576987 merged by Volans:
[operations/software/netbox-extras@master] dns: fix sub/24 IPv4 netmasks file generation

https://gerrit.wikimedia.org/r/576987

Change 577644 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: retrocompatibility with older pynetbox

https://gerrit.wikimedia.org/r/577644

Change 577644 merged by Volans:
[operations/software/netbox-extras@master] dns: retrocompatibility with older pynetbox

https://gerrit.wikimedia.org/r/577644

Change 578506 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] dns::auth: add DNS snippets generated from Netbox

https://gerrit.wikimedia.org/r/578506

Change 578531 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: new cookbook

https://gerrit.wikimedia.org/r/578531

Change 577528 merged by Volans:
[operations/software/netbox-extras@master] dns: add support for two-phase commit

https://gerrit.wikimedia.org/r/577528

Change 578925 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: fine tune snippet generation script

https://gerrit.wikimedia.org/r/578925

Mentioned in SAL (#wikimedia-operations) [2020-03-11T18:25:11Z] <volans> temporary disabled puppet on A:dns-auth to deploy g/578506 T233183

Change 578506 merged by Volans:
[operations/puppet@production] dns: add the Netbox driven DNS zonefile snippets

https://gerrit.wikimedia.org/r/578506

Change 578531 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: new cookbook

https://gerrit.wikimedia.org/r/578531

Change 579058 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: fix bug in dry-run mode

https://gerrit.wikimedia.org/r/579058

Change 578925 merged by Volans:
[operations/software/netbox-extras@master] dns: fine tune snippet generation script

https://gerrit.wikimedia.org/r/578925

Change 579058 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: fix bug in dry-run mode

https://gerrit.wikimedia.org/r/579058

Change 579239 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: fix CWD for runuser execution

https://gerrit.wikimedia.org/r/579239

Change 579239 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: fix CWD for runuser execution

https://gerrit.wikimedia.org/r/579239

Change 579242 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: add missing -- to runuser commands

https://gerrit.wikimedia.org/r/579242

Change 579242 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: add missing -- to runuser commands

https://gerrit.wikimedia.org/r/579242

Change 579249 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: add prefix to metadata

https://gerrit.wikimedia.org/r/579249

Change 579251 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: properly read metadata line

https://gerrit.wikimedia.org/r/579251

Change 579249 merged by Volans:
[operations/software/netbox-extras@master] dns: add prefix to metadata

https://gerrit.wikimedia.org/r/579249

Change 579251 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: properly read metadata line

https://gerrit.wikimedia.org/r/579251

Change 579254 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: exit early if no changes

https://gerrit.wikimedia.org/r/579254

Change 579254 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: exit early if no changes

https://gerrit.wikimedia.org/r/579254

Change 579271 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: fix metadata detection in output

https://gerrit.wikimedia.org/r/579271

Change 579271 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: fix metadata detection in output

https://gerrit.wikimedia.org/r/579271

Change 579336 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: improve logging

https://gerrit.wikimedia.org/r/579336

Change 579336 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: improve logging

https://gerrit.wikimedia.org/r/579336

Change 580954 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Remove host mgmt records for decommissioning hosts

https://gerrit.wikimedia.org/r/580954

Change 580955 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Remove all mgmt records for offline hosts

https://gerrit.wikimedia.org/r/580955

Change 580956 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Fix typos

https://gerrit.wikimedia.org/r/580956

Change 580956 merged by Dzahn:
[operations/dns@master] Fix typos

https://gerrit.wikimedia.org/r/580956

Change 580954 merged by Dzahn:
[operations/dns@master] Remove host mgmt records for decommissioning hosts

https://gerrit.wikimedia.org/r/580954

@BBlack @crusnov This is the script I use to compare the results P10716 both ways.

These is the output checking that all ops/dns repo records are present in the generated ones: P10718
This is the output checking that all netbox generated records are present in the operation/dns repo: P10717

The diff includes the changes in https://gerrit.wikimedia.org/r/c/operations/dns/+/580955 as if they were already merged.

I've opened T248035 to have the records for frack fixed.

Change 580955 merged by Volans:
[operations/dns@master] Remove all mgmt records for offline hosts

https://gerrit.wikimedia.org/r/580955

Change 583109 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: deploy the changes to gdnsd

https://gerrit.wikimedia.org/r/583109

In relation to https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/583109 I've just tested the command on a single host, this is the result:

root@dns4002:~# cd /srv/authdns/git && utils/deploy-check.py -g /srv/git/netbox_dns_snippets --deploy
Assembling and testing data in /tmp/dns-check.45j_ldfk
 -- Generating zonefiles from zone templates
 -- Processed 203 zones into directory /tmp/dns-check.45j_ldfk/zones
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
OK: No tabs
 -- Running zone_validator to check WMF rules
Summary of violations:
    W001|MISSING_IP_FOR_NAME_AND_PTR: 381
    W002|MISSING_PTR_FOR_NAME_AND_IP: 47
    W101|MISSING_ASSET_TAG: 262
    W103|MISSING_MGMT_FOR_NAME: 415
    W104|TOO_FEW_MGMT_NAMES: 394
    W105|TOO_MANY_PUBLIC_NAMES: 23
RESULT: 0 Errors, 1522 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.45j_ldfk
 -- Preflight checkconf is OK
Deploying from /tmp/dns-check.45j_ldfk to system dirs
 -- Descending to subdirectory: netbox
 -- Done with subdir: netbox
No action needed, zones and config files unchanged
OK

Change 583109 merged by Volans:
[operations/cookbooks@master] sre.dns.netbox: deploy the changes to gdnsd

https://gerrit.wikimedia.org/r/583109

Change 583676 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: pull the specific SHA1

https://gerrit.wikimedia.org/r/583676

Change 583676 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: pull the specific SHA1

https://gerrit.wikimedia.org/r/583676

Change 585545 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] mgmt: use netbox-generated data for ulsfo

https://gerrit.wikimedia.org/r/585545

crusnov moved this task from In Progress to Pending on the User-crusnov board.May 1 2020, 9:58 PM

Change 599948 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: add support for virtual machines

https://gerrit.wikimedia.org/r/599948

Change 599948 merged by Volans:
[operations/software/netbox-extras@master] dns: add support for virtual machines

https://gerrit.wikimedia.org/r/599948

Change 585545 merged by Volans:
[operations/dns@master] mgmt: use netbox-generated data for ulsfo

https://gerrit.wikimedia.org/r/585545

Mentioned in SAL (#wikimedia-operations) [2020-06-01T14:44:42Z] <volans> deploying ulsfo mgmt DNS records automatically generated by Netbox ( operations/dns/+/585545/ ) - T233183

Change 601434 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] mgmt: use netbox-generated data for eqsin mgmt

https://gerrit.wikimedia.org/r/601434

Change 601877 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: add support for primary IP generation

https://gerrit.wikimedia.org/r/601877

Change 601434 merged by Volans:
[operations/dns@master] mgmt: use netbox-generated data for eqsin mgmt

https://gerrit.wikimedia.org/r/601434

Mentioned in SAL (#wikimedia-operations) [2020-06-09T18:36:44Z] <volans> migrated mgmt DNS records in eqsin to the Netbox-generated records - T233183

Change 604136 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] mgmt: use netbox-generated data for esams mgmt

https://gerrit.wikimedia.org/r/604136

Change 604136 merged by Volans:
[operations/dns@master] mgmt: use netbox-generated data for esams mgmt

https://gerrit.wikimedia.org/r/604136

Mentioned in SAL (#wikimedia-operations) [2020-06-15T09:42:48Z] <volans> deploying esams mgmt DNS records automatically generated by Netbox ( operations/dns/+/604136/ ) - T233183

Change 601877 merged by Volans:
[operations/software/netbox-extras@master] scripts: complete interface automation generation

https://gerrit.wikimedia.org/r/601877

Change 605853 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: add esams to the mgmt migrated list

https://gerrit.wikimedia.org/r/605853

Change 605853 merged by Volans:
[operations/software/netbox-extras@master] scripts: add esams to the mgmt migrated list

https://gerrit.wikimedia.org/r/605853

@ayounsi @crusnov I've created mini-one-time script to automatically reserve the first 5 IP addresses in all relevant prefixes. Did a test run on af-netbox, the code and results are in P11559

LGTM! Especially for a one-time job.

Nice, this is what we pretty much had in mind, although in the future of course if we add more prefixes or change them we'll have to reserve them manually or whatever.

Nice, this is what we pretty much had in mind, although in the future of course if we add more prefixes or change them we'll have to reserve them manually or whatever.

Yes. We could make it a netbox custom script and re-run every time a new prefix is created, if it matches the criteria as it's idempotent.

Mentioned in SAL (#wikimedia-operations) [2020-06-19T08:45:47Z] <volans> backup netbox and run one-time script to reserve first IPs on all infra prefixes on Netbox - T233183

I've run the script in production, you can see the output of the script in P11603 and the results in Netbox in two ways:

In case of any immediate issue there is a backup on netboxdb1001 taken right before running the script in /srv/postgres-backup/volans.pre.reserve.ips.T233183.psql-all-dbs-20200619.sql.gz.bak.

I've created this one-off script and run it on the af-netbox test instance to cleanup ifaces and addresses from existing offline devices: P11631

Mentioned in SAL (#wikimedia-operations) [2020-06-22T22:12:34Z] <volans> cleanup interfaces and addresses in Netbox for offline servers - T233183

Got it reviewed by @crusnov, run on production with P11632
In case of any immediate issue there is a backup on netboxdb1001 taken right before running the script in /srv/postgres-backup//srv/postgres-backup/volans.pre.cleanup.offline.T233183.psql-all-dbs-20200622.sql.gz.bak

Change 607741 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] mgmt: netbox-generated data for frack mgmt codfw

https://gerrit.wikimedia.org/r/607741

Change 607741 merged by CRusnov:
[operations/dns@master] mgmt: netbox-generated data for frack mgmt codfw

https://gerrit.wikimedia.org/r/607741

Mentioned in SAL (#wikimedia-operations) [2020-07-09T17:42:35Z] <chaomodus> codfw frack management dns automation deployment complete T233183

Change 612472 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] mgmt: netbox-generated data for frack mgmt eqiad

https://gerrit.wikimedia.org/r/612472

Change 612472 merged by Volans:
[operations/dns@master] mgmt: netbox-generated data for frack mgmt eqiad

https://gerrit.wikimedia.org/r/612472

Change 615668 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] mgmt: netbox-generated data for mgmt codfw

https://gerrit.wikimedia.org/r/615668

Change 617119 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: skip Netbox addresses without DNS name

https://gerrit.wikimedia.org/r/617119

Change 617119 merged by Volans:
[operations/software/netbox-extras@master] dns: check that primary addresses have DNS names

https://gerrit.wikimedia.org/r/617119

Mentioned in SAL (#wikimedia-operations) [2020-07-29T13:55:34Z] <volans> migrating *all* codfw mgmt DNS records to the autogenerated ones via Netbox - T233183

Change 615668 merged by Volans:
[operations/dns@master] mgmt: netbox-generated data for mgmt codfw

https://gerrit.wikimedia.org/r/615668

Change 617147 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: codfw migrated to Netbox

https://gerrit.wikimedia.org/r/617147

Change 617147 merged by Volans:
[operations/software/netbox-extras@master] scripts: codfw migrated to Netbox

https://gerrit.wikimedia.org/r/617147

Change 617509 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] mgmt: netbox-generated data for mgmt eqiad

https://gerrit.wikimedia.org/r/617509

Change 619015 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] mgmt codfw: migrated Papaul's IP to Netbox

https://gerrit.wikimedia.org/r/619015

Change 619015 merged by Volans:
[operations/dns@master] mgmt codfw: migrated Papaul's IP to Netbox

https://gerrit.wikimedia.org/r/619015

Mentioned in SAL (#wikimedia-operations) [2020-08-11T10:39:28Z] <volans> migrating *all* eqiad mgmt DNS records to the autogenerated ones via Netbox - T233183

Change 617509 merged by Volans:
[operations/dns@master] mgmt: netbox-generated data for mgmt eqiad

https://gerrit.wikimedia.org/r/617509

Volans closed this task as Resolved.Aug 11 2020, 11:36 AM
Volans claimed this task.

All management records are now generated via Netbox, related wikitech documentation updated (both the DNS/Netbox page and the Server Lifecycle one). Resolving