As spoken on IRC, if there is a clean cut to identify when we can use /24 in terms of Netbox data of the prefixes and such, and if it's ok to loose some of the flexibility we have right now to migrate gradually some things it's not a problem for me to adapt the generation script.

If possible though I'll strongly advice if we have to do this to do it immediately, before we migrate the 2 main DCs as the transition is more complicated than the change itself.

If instead we keep the current per-prefix approach we can easily add some additional checks in the sre.dns.netbox cookbook such as:

• for each modified auto-generated file check that they are $INCLUDEd in the dns repo (this can be done only after we've migrated everything with maybe a whitelist)
• if there is any newly added file add a message to the user that a patch in the dns repo is needed to include the new file(s)
• if there is any removed file pause and ask the user to make a patch to remove the INCLUDEs from the manual dns repo before proceeding with the push, to prevent the deploy to fail at all.

and something to the dns repo CI too:

• check that all $INCLUDE files are unique (no duplicates)

After a chat with @ayounsi I tried the approach that if the IP prefixlen is > 24 instead of picking the smallest prefix we try to get the first prefix above with status container and use that instead for deciding the zonefile.

The actual code in case we go for this option will of course have a temporary patch that will generate both records to allow to switch the already migrated DCs from the previous INCLUDEs to those more compact INCLUDEs and it will be removed after that.

For now I've just tested the change itself that is easy to see as a diff, and pasted it here: P12937

As you can see there are some issues where the first container is larger than a /24. Are those ok? What filename should we use for them? The same syntax of smaller prefixes, hence 0-23.152.80.208.in-addr.arpa for 208.80.152.0/23?

For IPv4 you'll have to break up the <24 cases into /24 containers somehow, because they DNS zones themselves are /24 in that case. Perhaps we have to make some new containers in netbox, which represent the DNS-level abstraction, in this case?

Yeah I was chatting on IRC with Arzhel and he was suggesting to force them to /24 anyway, even if we don't have the prefix in Netbox.

I did that change and update the paste P12937 with the result.

Those are the file changes:

Created

128-27.166.102.103.in-addr.arpa
153.80.208.in-addr.arpa
154.80.208.in-addr.arpa
26.35.198.in-addr.arpa

Renamed

rename from 96-27.0.195.10.in-addr.arpa to 0.195.10.in-addr.arpa
rename from 240-28.152.80.208.in-addr.arpa to 152.80.208.in-addr.arpa
rename from 96-27.155.80.208.in-addr.arpa to 155.80.208.in-addr.arpa
rename from 0-28.166.102.103.in-addr.arpa to 166.102.103.in-addr.arpa
rename from 0-30.4.132.10.in-addr.arpa to 4.132.10.in-addr.arpa
rename from 0-30.4.20.10.in-addr.arpa to 4.20.10.in-addr.arpa
rename from 192-26.40.64.10.in-addr.arpa to 40.64.10.in-addr.arpa
rename from 248-29.59.15.185.in-addr.arpa to 59.15.185.in-addr.arpa

Deleted

4-30.4.132.10.in-addr.arpa
4-30.4.20.10.in-addr.arpa
0-30.0.66.10.in-addr.arpa
4-30.0.66.10.in-addr.arpa
128-30.166.102.103.in-addr.arpa
132-31.166.102.103.in-addr.arpa
134-31.166.102.103.in-addr.arpa
136-31.166.102.103.in-addr.arpa
138-31.166.102.103.in-addr.arpa
140-31.166.102.103.in-addr.arpa
142-31.166.102.103.in-addr.arpa
144-31.166.102.103.in-addr.arpa
16-28.166.102.103.in-addr.arpa
224-27.166.102.103.in-addr.arpa
0-28.26.35.198.in-addr.arpa
196-31.26.35.198.in-addr.arpa
198-31.26.35.198.in-addr.arpa
200-31.26.35.198.in-addr.arpa
204-31.26.35.198.in-addr.arpa
208-31.26.35.198.in-addr.arpa
210-31.26.35.198.in-addr.arpa
224-29.26.35.198.in-addr.arpa
240-28.26.35.198.in-addr.arpa
96-27.26.35.198.in-addr.arpa
0-27.153.80.208.in-addr.arpa
184-29.153.80.208.in-addr.arpa
200-31.153.80.208.in-addr.arpa
202-31.153.80.208.in-addr.arpa
204-31.153.80.208.in-addr.arpa
206-31.153.80.208.in-addr.arpa
208-31.153.80.208.in-addr.arpa
210-31.153.80.208.in-addr.arpa
212-31.153.80.208.in-addr.arpa
214-31.153.80.208.in-addr.arpa
216-31.153.80.208.in-addr.arpa
218-31.153.80.208.in-addr.arpa
220-31.153.80.208.in-addr.arpa
222-31.153.80.208.in-addr.arpa
224-27.153.80.208.in-addr.arpa
32-27.153.80.208.in-addr.arpa
64-27.153.80.208.in-addr.arpa
96-27.153.80.208.in-addr.arpa
0-26.154.80.208.in-addr.arpa
128-26.154.80.208.in-addr.arpa
192-30.154.80.208.in-addr.arpa
196-30.154.80.208.in-addr.arpa
200-31.154.80.208.in-addr.arpa
202-31.154.80.208.in-addr.arpa
204-31.154.80.208.in-addr.arpa
206-31.154.80.208.in-addr.arpa
208-31.154.80.208.in-addr.arpa
210-31.154.80.208.in-addr.arpa
220-31.154.80.208.in-addr.arpa
224-27.154.80.208.in-addr.arpa
64-26.154.80.208.in-addr.arpa
64-28.155.80.208.in-addr.arpa
88-29.155.80.208.in-addr.arpa
0-25.174.198.91.in-addr.arpa
128-28.174.198.91.in-addr.arpa
192-27.174.198.91.in-addr.arpa
228-31.174.198.91.in-addr.arpa
240-31.174.198.91.in-addr.arpa
248-31.174.198.91.in-addr.arpa
250-31.174.198.91.in-addr.arpa
252-31.174.198.91.in-addr.arpa
254-31.174.198.91.in-addr.arpa

Change 632574 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: consolidate reverse zone files

https://gerrit.wikimedia.org/r/632574

Change 632952 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: consolidate reverse zone files (part 2)

https://gerrit.wikimedia.org/r/632952

Change 632953 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] netbox: move $INCLUDEs to the consolidated files

https://gerrit.wikimedia.org/r/632953

The previous approach was not working well because I found corner cases where we were consolidating into the same file different subnets for which one is managed by Netbox and one not and will probably never be (like frack vs frack mgmt).

Moved to a smaller approach where we consolidate only /30 and /31 prefixes into their parent. This reduces the impact a lot, affecting basically only network devices almost but at the same time allow to reduce by ~48 the number of reverse zonefiles down to ~168.

Mentioned in SAL (#wikimedia-operations) [2020-10-08T20:43:34Z] <volans> deploying Netbox DNS zone consolidation - T264273

Change 632574 merged by Volans:
[operations/software/netbox-extras@master] dns: consolidate reverse zone files (part 1)

https://gerrit.wikimedia.org/r/632574

Change 632953 merged by Volans:
[operations/dns@master] netbox: move $INCLUDEs to the consolidated files

https://gerrit.wikimedia.org/r/632953

Change 632952 merged by Volans:
[operations/software/netbox-extras@master] dns: consolidate reverse zone files (part 2)

https://gerrit.wikimedia.org/r/632952