Page MenuHomePhabricator

Improve how we generate DNS entries from Netbox
Open, LowPublic

Description

Current Setup

Right now the sre.dns.netbox cookbook generates numerous different files which ultimaty go into /etc/gdnsd/zones/netbox/ on auth dns servers. Each of these needs to have an INCLUDE statement pointing to it in the static zone files we create in the dns repo. This is extremely cumbersome, tedious and error-prone, especially for IPv6 reverses where we have a separate file for every single link on the network (as each have their own /64 subnet).

Right now, for instance, our zone files have entries like this:

cmooney@dns3003:/etc/gdnsd/zones$ tail -7 0.8.c.e.2.0.a.2.ip6.arpa 
; cr2-magru <-> asw1-b4-magru (2a02:ec80:700:fe09::/64)
$ORIGIN 9.0.e.f.0.0.7.0.@Z
$INCLUDE netbox/9.0.e.f.0.0.7.0.0.8.c.e.2.0.a.2.ip6.arpa

; Magru network loopbacks (2a02:ec80:700:ffff::/64)
$ORIGIN f.f.f.f.0.0.7.0.@Z
$INCLUDE netbox/f.f.f.f.0.0.7.0.0.8.c.e.2.0.a.2.ip6.arpa

The contents of each of the included files, generated by Netbox, only have the PTR (or A / AAAA) records that go under that $ORIGIN:

cmooney@dns3003:/etc/gdnsd/zones$ cat ./netbox/9.0.e.f.0.0.7.0.0.8.c.e.2.0.a.2.ip6.arpa 
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR et-0-0-2.cr2-magru.wikimedia.org.
2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR et-0-0-50.asw1-b4-magru.wikimedia.org.
cmooney@dns3003:/etc/gdnsd/zones$ cat ./netbox/f.f.f.f.0.0.7.0.0.8.c.e.2.0.a.2.ip6.arpa
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cr1-magru.wikimedia.org.
Alternate Approach

What I would propose is we change how we do things as follows:

  • For each zone file we are authoritative for we should have a single INCLUDE statement for netbox entries
  • The INCLUDE should be at the zone 'apex' (i.e. $ORIGIN @Z)
  • The single file it points to should have all the records from Netbox that belong to that zone
  • We define all labels in the snippet files relative to the zone itself, and use no further $ORIGIN directives

/etc/gdnsd/zones/0.8.c.e.2.0.a.2.ip6.arpa:

@ 1H IN SOA ns0.wikimedia.org. hostmaster.wikimedia.org. 2024041716 12H 2H 2W 1H

@                       1D      IN NS   ns0.wikimedia.org.
@                       1D      IN NS   ns1.wikimedia.org.
@                       1D      IN NS   ns2.wikimedia.org.

; Netbox generated entries
$INCLUDE netbox/0.8.c.e.2.0.a.2.ip6.arpa @Z

; Static entries
$ORIGIN x xx x x
yyy   IN   PTR    blah.wikimedia.org.

/etc/gdnsd/zones/netbox/0.8.c.e.2.0.a.2.ip6.arpa:

$ORIGIN @Z
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.9.0.e.f.0.0.7.0 IN    PTR    et-0-0-2.cr2-magru.wikimedia.org.
2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.9.0.e.f.0.0.7.0 IN    PTR    et-0-0-50.asw1-b4-magru.wikimedia.org.
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.f.f.f.f.0.0.7.0 IN    PTR    cr1-magru.wikimedia.org.

Event Timeline

cmooney triaged this task as Low priority.

Another question I think is "do we still have to go through text files ?"
It made sens for back in the time when we were manually editing the configuration, and for the few places we still do, but it seems sub-optimal to go from Netbox database to text file to gdnsd.
Probably too simplistic, but could we generate a raw list of IP/FQDN from Netbox, and feed it to gDNSd without having to care about PTR and zones structures ?

Another question I think is "do we still have to go through text files ?"
It made sens for back in the time when we were manually editing the configuration, and for the few places we still do, but it seems sub-optimal to go from Netbox database to text file to gdnsd.
Probably too simplistic, but could we generate a raw list of IP/FQDN from Netbox, and feed it to gDNSd without having to care about PTR and zones structures ?

I guess the advantage of writing out the raw FQDNs / IPs is that the Netbox cookbook would not need to understand what zone each record needs to go into. It could just dump one big file (or push it somehow with an API or something).

But ultimately we'd still need to write the code that took those names, and did write them out in zone file format into the correct files. So it seems we'd just be moving the work around here? I don't really get what the extra layer of indirection of having our cookbook generate the interim data, which needs to be further processed (by another piece of middleware or gdnsd itself), brings?

I've can imagine various different approaches to what we are doing right now tbh, of varying levels of sophistication. Without going into them all my gut feeling is generating the zone file content directly from Netbox remains the most straightforward and robust.

So I've been meaning to look at this for ages and while how to generate the records was clear to me, how to update the existing cookbook and snippet generation script wasn't so obvious.

So I sort of cut my losses and made a script that generates the snippets, which can hopefully serve as the basis for discussing how best to proceed:

https://github.com/topranks/random_wmf/tree/main/netbox_dns

If this looks somewhat acceptable to people we could start thinking about next steps.

When I have time I will also try to validate the data being generated is correct by:

  • Setting up a test authdns server on a VM, container or similar
  • Loading zone files from this script in it
  • Doing a lookup on every entry in this "test" authdns server, and compare to the result if the lookup is done to nsX.wikimedia.org

Which will hopefully verify everything is consistent. In terms of the wider work to integrate with Netbox and get data onto our authdns hosts I will need to work with others on the team who are more familiar.

Which will hopefully verify everything is consistent. In terms of the wider work to integrate with Netbox and get data onto our authdns hosts I will need to work with others on the team who are more familiar.

Very happy to help with this of course (on the authdns side of things) and thanks for working on it! If you need someone to review the patches, please add me to it.

cmooney updated the task description. (Show Details)
cmooney updated the task description. (Show Details)

Seems like a net win to me. Reduces some error-prone process stuff and makes life simpler!

Also, probably the way to standardize this for sanity (avoiding ORIGIN mistakes on both ends) is to follow some simple rules that:

  1. Every one of the new include files starts with an internal $ORIGIN @Z
  2. That the include statements for them look like: $INCLUDE netbox/whatever @Z

Edited after I remembered the include-with-origin syntax

cmooney updated the task description. (Show Details)

Also, probably the way to standardize this for sanity (avoiding ORIGIN mistakes on both ends) is to follow some simple rules that:

  1. Every one of the new include files starts with an internal $ORIGIN @Z
  2. That the include statements for them look like: $INCLUDE netbox/whatever @Z

Edited after I remembered the include-with-origin syntax

Thanks that makes sense, I've updated the task description to match that pattern.

Change #1163355 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Add new reposync repo called 'netbox-dns-records'

https://gerrit.wikimedia.org/r/1163355

Change #1163355 merged by Cathal Mooney:

[operations/puppet@production] Add new reposync repo called 'netbox-dns'

https://gerrit.wikimedia.org/r/1163355

Change #1163382 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Netbox hosts: add netbox-dns reposync repo so it is available

https://gerrit.wikimedia.org/r/1163382

Change #1163436 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Netbox hosts: ensure reposync repos are set up to match cumin hosts

https://gerrit.wikimedia.org/r/1163436

Change #1163436 abandoned by Cathal Mooney:

[operations/puppet@production] Netbox hosts: ensure reposync repos are set up to match cumin hosts

Reason:

profile::spicerack::reposync::repos is only defined for the automation role and can't be read here.

https://gerrit.wikimedia.org/r/1163436

Change #1163436 restored by Cathal Mooney:

[operations/puppet@production] Netbox hosts: ensure reposync repos are set up to match cumin hosts

https://gerrit.wikimedia.org/r/1163436

Change #1163382 abandoned by Cathal Mooney:

[operations/puppet@production] Netbox hosts: add netbox-dns reposync repo so it is available

Reason:

Will do this a different way instead

https://gerrit.wikimedia.org/r/1163382

Change #1163690 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:dns::auth::netbox Netbox DNS zones file sync

https://gerrit.wikimedia.org/r/1163690

Change #1163436 merged by Cathal Mooney:

[operations/puppet@production] Netbox hosts: ensure reposync repos are set up to match cumin hosts

https://gerrit.wikimedia.org/r/1163436

Change #1163690 merged by Slyngshede:

[operations/puppet@production] P:dns::auth::netbox Netbox DNS zones file sync

https://gerrit.wikimedia.org/r/1163690

Change #1163717 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:dns::auth::netbox_dns_records fix branch name

https://gerrit.wikimedia.org/r/1163717

Change #1163717 merged by Slyngshede:

[operations/puppet@production] P:dns::auth::netbox_dns_records fix branch name

https://gerrit.wikimedia.org/r/1163717

Change #1163766 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Authdns: add profile to role to clone new repo with netbox dns RRs

https://gerrit.wikimedia.org/r/1163766

Change #1163813 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Sretest: remove temporary additions testing dns repo stuff

https://gerrit.wikimedia.org/r/1163813

Change #1163813 merged by Cathal Mooney:

[operations/puppet@production] Sretest: remove temporary additions testing dns repo stuff

https://gerrit.wikimedia.org/r/1163813

Change #1163766 merged by Cathal Mooney:

[operations/puppet@production] Authdns: clone new netbox-generated DNS records repo

https://gerrit.wikimedia.org/r/1163766

Change #1164124 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] Add new Netbox records repo

https://gerrit.wikimedia.org/r/1164124

Change #1163318 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/cookbooks@master] sre.dns.netbox-records cookbook

https://gerrit.wikimedia.org/r/1163318