Page MenuHomePhabricator

Automate generation of Management DNS records from Netbox
Open, MediumPublic

Description

Get to the "testing" phase of automated generation:

  • Read Netbox API
  • Dump includable records and reverse records for Management interfaces
  • Test and verify produced records against manually maintained records

Details

Related Gerrit Patches:
operations/cookbooks : mastersre.dns.netbox: pull the specific SHA1
operations/cookbooks : mastersre.dns.netbox: deploy the changes to gdnsd
operations/dns : masterRemove all mgmt records for offline hosts
operations/dns : masterRemove host mgmt records for decommissioning hosts
operations/dns : masterFix typos
operations/cookbooks : mastersre.dns.netbox: improve logging
operations/cookbooks : mastersre.dns.netbox: fix metadata detection in output
operations/cookbooks : mastersre.dns.netbox: exit early if no changes
operations/cookbooks : mastersre.dns.netbox: properly read metadata line
operations/software/netbox-extras : masterdns: add prefix to metadata
operations/cookbooks : mastersre.dns.netbox: add missing -- to runuser commands
operations/cookbooks : mastersre.dns.netbox: fix CWD for runuser execution
operations/cookbooks : mastersre.dns.netbox: fix bug in dry-run mode
operations/software/netbox-extras : masterdns: fine tune snippet generation script
operations/cookbooks : mastersre.dns.netbox: new cookbook
operations/puppet : productiondns: add the Netbox driven DNS zonefile snippets
operations/software/netbox-extras : masterdns: add support for two-phase commit
operations/software/netbox-extras : masterdns: retrocompatibility with older pynetbox
operations/software/netbox-extras : masterdns: fix sub/24 IPv4 netmasks file generation
operations/software/netbox-extras : masterdns: convert records management in classes
operations/software/netbox-extras : masterdns: convert Netbox data gathering into a class
operations/software/netbox-extras : masterdns: manage separately servers from other devices
operations/software/netbox-extras : masterdns: manage also devices in Inventory state
operations/software/netbox-extras : masterdns: sort records by the rightmost part
operations/software/netbox-extras : masterdns: generate correct zone name in all cases
operations/software/netbox-extras : masterdns: include all IP addresses with FQDN
operations/software/netbox-extras : masterdns: make netbox API query backward compatible
operations/puppet : productionnetbox: fix path for dns snippets repo
operations/puppet : productionnetbox: add dns.cfg configuration file
operations/software/netbox-extras : masterdns: update default config file path
operations/software/netbox-extras : masterdns: handle push failures
operations/software/netbox-extras : masterdns: add batch mode to generate snippets script
operations/puppet : productionnetbox: fix variable name in template
operations/puppet : productionnetbox: fix path for file
operations/puppet : productionnetbox: create also parent directory
operations/puppet : productionnetbox: Add automation git machinery
operations/dns : masterFix typo for netbox-exports
operations/dns : masterAdd netbox-exports.w.o record as CNAME of netbox
operations/software/netbox-extras : masterdns: generate DNS snippets from Netbox
integration/config : masterSetup CI for operations/software/netbox-extras
operations/puppet : productionnetbox: Expose automated DNS repository for web access
operations/software/netbox-deploy : masterAdd script to generate DNS records from Netbox
integration/config : masterSetup CI for operations/software/netbox-extra
operations/puppet : productionnetbox: Setup automated DNS generation
operations/puppet : productionprofile::authdns: Add automation framework

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 551948 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] netbox: Expose automated DNS repository for web access

https://gerrit.wikimedia.org/r/551948

After a conversation with @Volans an extended ask is having the generator able to add and remove files (eg, override completely the contents of repository, if necessary). This is a bit of an extension of the workflow I'd envisioned but I shall be implementing that now.

Change 554543 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: generate DNS snippets from Netbox

https://gerrit.wikimedia.org/r/554543

Change 554560 had a related patch set uploaded (by Volans; owner: Volans):
[integration/config@master] Setup CI for operations/software/netbox-extra

https://gerrit.wikimedia.org/r/554560

Change 554560 merged by jenkins-bot:
[integration/config@master] Setup CI for operations/software/netbox-extra

https://gerrit.wikimedia.org/r/554560

Mentioned in SAL (#wikimedia-releng) [2019-12-04T16:49:01Z] <James_F> Zuul: Add CI for operations/software/netbox-extra T233183

Change 539013 abandoned by CRusnov:
Add script to generate DNS records from Netbox

Reason:
THis project is moved to netbox-extra.

https://gerrit.wikimedia.org/r/539013

Change 555715 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] netbox: Add automation git machinery

https://gerrit.wikimedia.org/r/555715

Change 551948 abandoned by CRusnov:
netbox: Expose automated DNS repository for web access

Reason:
this will be superseded by https://gerrit.wikimedia.org/r/c/operations/puppet/ /555715

https://gerrit.wikimedia.org/r/551948

Change 556228 had a related patch set uploaded (by Volans; owner: Volans):
[integration/config@master] Setup CI for operations/software/netbox-extras

https://gerrit.wikimedia.org/r/556228

Change 556228 merged by jenkins-bot:
[integration/config@master] Setup CI for operations/software/netbox-extras

https://gerrit.wikimedia.org/r/556228

Change 554543 merged by Volans:
[operations/software/netbox-extras@master] dns: generate DNS snippets from Netbox

https://gerrit.wikimedia.org/r/554543

Change 559638 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: add batch mode to generate snippets script

https://gerrit.wikimedia.org/r/559638

Change 559639 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: handle push failures

https://gerrit.wikimedia.org/r/559639

Change 559860 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Add netbox-exports.w.o record as CNAME of netbox

https://gerrit.wikimedia.org/r/559860

Change 559860 merged by Volans:
[operations/dns@master] Add netbox-exports.w.o record as CNAME of netbox

https://gerrit.wikimedia.org/r/559860

Change 559862 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Fix typo for netbox-exports

https://gerrit.wikimedia.org/r/559862

Change 559862 merged by Volans:
[operations/dns@master] Fix typo for netbox-exports

https://gerrit.wikimedia.org/r/559862

Mentioned in SAL (#wikimedia-operations) [2019-12-20T14:38:43Z] <volans> temporarily disable puppet on netbox[12]001 to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/555715 - T233183

Change 555715 merged by Volans:
[operations/puppet@production] netbox: Add automation git machinery

https://gerrit.wikimedia.org/r/555715

Change 559867 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: create also parent directory

https://gerrit.wikimedia.org/r/559867

Change 559867 merged by Volans:
[operations/puppet@production] netbox: create also parent directory

https://gerrit.wikimedia.org/r/559867

Change 559875 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: fix path for file

https://gerrit.wikimedia.org/r/559875

Change 559875 merged by Volans:
[operations/puppet@production] netbox: fix path for file

https://gerrit.wikimedia.org/r/559875

Change 559882 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: fix variable name in template

https://gerrit.wikimedia.org/r/559882

Change 559882 merged by Volans:
[operations/puppet@production] netbox: fix variable name in template

https://gerrit.wikimedia.org/r/559882

Change 559897 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: add dns.cfg configuration file

https://gerrit.wikimedia.org/r/559897

Change 559638 merged by Volans:
[operations/software/netbox-extras@master] dns: add batch mode to generate snippets script

https://gerrit.wikimedia.org/r/559638

Change 559639 merged by Volans:
[operations/software/netbox-extras@master] dns: handle push failures

https://gerrit.wikimedia.org/r/559639

Change 559899 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: update default config file path

https://gerrit.wikimedia.org/r/559899

Change 559899 merged by Volans:
[operations/software/netbox-extras@master] dns: update default config file path

https://gerrit.wikimedia.org/r/559899

Change 559897 merged by Volans:
[operations/puppet@production] netbox: add dns.cfg configuration file

https://gerrit.wikimedia.org/r/559897

Change 559912 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] netbox: fix path for dns snippets repo

https://gerrit.wikimedia.org/r/559912

Change 559913 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: make netbox API query backward compatible

https://gerrit.wikimedia.org/r/559913

Change 559912 merged by Volans:
[operations/puppet@production] netbox: fix path for dns snippets repo

https://gerrit.wikimedia.org/r/559912

Change 559913 merged by Volans:
[operations/software/netbox-extras@master] dns: make netbox API query backward compatible

https://gerrit.wikimedia.org/r/559913

Change 561601 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: include all IP addresses with FQDN

https://gerrit.wikimedia.org/r/561601

Change 561602 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: generate correct zone name in all cases

https://gerrit.wikimedia.org/r/561602

Change 561603 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: sort records by the rightmost part

https://gerrit.wikimedia.org/r/561603

Change 561917 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: manage also devices in Inventory state

https://gerrit.wikimedia.org/r/561917

Change 561918 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: manage separately servers from other devices

https://gerrit.wikimedia.org/r/561918

Change 561601 merged by Volans:
[operations/software/netbox-extras@master] dns: include all IP addresses with FQDN

https://gerrit.wikimedia.org/r/561601

Change 561602 merged by Volans:
[operations/software/netbox-extras@master] dns: generate correct zone name in all cases

https://gerrit.wikimedia.org/r/561602

Change 561603 merged by Volans:
[operations/software/netbox-extras@master] dns: sort records by the rightmost part

https://gerrit.wikimedia.org/r/561603

Change 561917 merged by Volans:
[operations/software/netbox-extras@master] dns: manage also devices in Inventory state

https://gerrit.wikimedia.org/r/561917

Change 561918 merged by Volans:
[operations/software/netbox-extras@master] dns: manage separately servers from other devices

https://gerrit.wikimedia.org/r/561918

crusnov moved this task from Backlog to In Progress on the User-crusnov board.Feb 14 2020, 8:14 PM

Change 576985 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: convert Netbox data gathering into a class

https://gerrit.wikimedia.org/r/576985

Change 576986 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: convert records management in classes

https://gerrit.wikimedia.org/r/576986

Change 576987 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: fix sub/24 IPv4 netmasks file generation

https://gerrit.wikimedia.org/r/576987

Change 577528 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: add support for two-phase commit

https://gerrit.wikimedia.org/r/577528

Change 576985 merged by Volans:
[operations/software/netbox-extras@master] dns: convert Netbox data gathering into a class

https://gerrit.wikimedia.org/r/576985

Change 576986 merged by Volans:
[operations/software/netbox-extras@master] dns: convert records management in classes

https://gerrit.wikimedia.org/r/576986

Change 576987 merged by Volans:
[operations/software/netbox-extras@master] dns: fix sub/24 IPv4 netmasks file generation

https://gerrit.wikimedia.org/r/576987

Change 577644 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: retrocompatibility with older pynetbox

https://gerrit.wikimedia.org/r/577644

Change 577644 merged by Volans:
[operations/software/netbox-extras@master] dns: retrocompatibility with older pynetbox

https://gerrit.wikimedia.org/r/577644

Change 578506 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] dns::auth: add DNS snippets generated from Netbox

https://gerrit.wikimedia.org/r/578506

Change 578531 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: new cookbook

https://gerrit.wikimedia.org/r/578531

Change 577528 merged by Volans:
[operations/software/netbox-extras@master] dns: add support for two-phase commit

https://gerrit.wikimedia.org/r/577528

Change 578925 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: fine tune snippet generation script

https://gerrit.wikimedia.org/r/578925

Mentioned in SAL (#wikimedia-operations) [2020-03-11T18:25:11Z] <volans> temporary disabled puppet on A:dns-auth to deploy g/578506 T233183

Change 578506 merged by Volans:
[operations/puppet@production] dns: add the Netbox driven DNS zonefile snippets

https://gerrit.wikimedia.org/r/578506

Change 578531 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: new cookbook

https://gerrit.wikimedia.org/r/578531

Change 579058 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: fix bug in dry-run mode

https://gerrit.wikimedia.org/r/579058

Change 578925 merged by Volans:
[operations/software/netbox-extras@master] dns: fine tune snippet generation script

https://gerrit.wikimedia.org/r/578925

Change 579058 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: fix bug in dry-run mode

https://gerrit.wikimedia.org/r/579058

Change 579239 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: fix CWD for runuser execution

https://gerrit.wikimedia.org/r/579239

Change 579239 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: fix CWD for runuser execution

https://gerrit.wikimedia.org/r/579239

Change 579242 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: add missing -- to runuser commands

https://gerrit.wikimedia.org/r/579242

Change 579242 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: add missing -- to runuser commands

https://gerrit.wikimedia.org/r/579242

Change 579249 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: add prefix to metadata

https://gerrit.wikimedia.org/r/579249

Change 579251 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: properly read metadata line

https://gerrit.wikimedia.org/r/579251

Change 579249 merged by Volans:
[operations/software/netbox-extras@master] dns: add prefix to metadata

https://gerrit.wikimedia.org/r/579249

Change 579251 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: properly read metadata line

https://gerrit.wikimedia.org/r/579251

Change 579254 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: exit early if no changes

https://gerrit.wikimedia.org/r/579254

Change 579254 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: exit early if no changes

https://gerrit.wikimedia.org/r/579254

Change 579271 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: fix metadata detection in output

https://gerrit.wikimedia.org/r/579271

Change 579271 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: fix metadata detection in output

https://gerrit.wikimedia.org/r/579271

Change 579336 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: improve logging

https://gerrit.wikimedia.org/r/579336

Change 579336 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: improve logging

https://gerrit.wikimedia.org/r/579336

Change 580954 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Remove host mgmt records for decommissioning hosts

https://gerrit.wikimedia.org/r/580954

Change 580955 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Remove all mgmt records for offline hosts

https://gerrit.wikimedia.org/r/580955

Change 580956 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Fix typos

https://gerrit.wikimedia.org/r/580956

Change 580956 merged by Dzahn:
[operations/dns@master] Fix typos

https://gerrit.wikimedia.org/r/580956

Change 580954 merged by Dzahn:
[operations/dns@master] Remove host mgmt records for decommissioning hosts

https://gerrit.wikimedia.org/r/580954

@BBlack @crusnov This is the script I use to compare the results P10716 both ways.

These is the output checking that all ops/dns repo records are present in the generated ones: P10718
This is the output checking that all netbox generated records are present in the operation/dns repo: P10717

The diff includes the changes in https://gerrit.wikimedia.org/r/c/operations/dns/+/580955 as if they were already merged.

I've opened T248035 to have the records for frack fixed.

Change 580955 merged by Volans:
[operations/dns@master] Remove all mgmt records for offline hosts

https://gerrit.wikimedia.org/r/580955

Change 583109 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: deploy the changes to gdnsd

https://gerrit.wikimedia.org/r/583109

In relation to https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/583109 I've just tested the command on a single host, this is the result:

root@dns4002:~# cd /srv/authdns/git && utils/deploy-check.py -g /srv/git/netbox_dns_snippets --deploy
Assembling and testing data in /tmp/dns-check.45j_ldfk
 -- Generating zonefiles from zone templates
 -- Processed 203 zones into directory /tmp/dns-check.45j_ldfk/zones
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
OK: No tabs
 -- Running zone_validator to check WMF rules
Summary of violations:
    W001|MISSING_IP_FOR_NAME_AND_PTR: 381
    W002|MISSING_PTR_FOR_NAME_AND_IP: 47
    W101|MISSING_ASSET_TAG: 262
    W103|MISSING_MGMT_FOR_NAME: 415
    W104|TOO_FEW_MGMT_NAMES: 394
    W105|TOO_MANY_PUBLIC_NAMES: 23
RESULT: 0 Errors, 1522 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.45j_ldfk
 -- Preflight checkconf is OK
Deploying from /tmp/dns-check.45j_ldfk to system dirs
 -- Descending to subdirectory: netbox
 -- Done with subdir: netbox
No action needed, zones and config files unchanged
OK

Change 583109 merged by Volans:
[operations/cookbooks@master] sre.dns.netbox: deploy the changes to gdnsd

https://gerrit.wikimedia.org/r/583109

Change 583676 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: pull the specific SHA1

https://gerrit.wikimedia.org/r/583676