AIUI we have a regular job that sends a diff between the expected Homer config and the live running config to Netops. However as the Capirca Netbox script is only ran manually, so changes to the generated firewall config can go unnoticed for a while. For example on T361537 I also had to deploy unrelated changes that replaced the apt hosts and added a contint one. Could we run the Capirca Netbox script regularly on a timer to have these pending changes noticed more regularly?
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | ayounsi | T361549 Automatically run Capirca Netbox script regularly | |||
| Duplicate | None | T358339 Netbox: capirca.getHosts script runs into timeout | |||
| Resolved | ayounsi | T336275 Upgrade Netbox to 4.x |
Event Timeline
Comment Actions
Thanks for the task. I was thinking of either a timer or using Netbox's hooks to only run it when relevant changes are done.
This however always brings the question of where the pending changes should wait. Running the script automatically will result in changes pending to be pushed to network devices.
Running it on a timer is also not possible until T358339: Netbox: capirca.getHosts script runs into timeout is done.
Comment Actions
Change #1218209 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/software/netbox-extras@master] [WIP] Capirca: only show diff when running in "non-commit"
Comment Actions
Change #1218739 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/software/homer@master] Capirca: look for the most recent completed run
Comment Actions
Once the two patches above are deployed, comes the question on how to run it regularly.
There are 2 possible options :
- Timer (like other scripts or reports)
- Event Rule
The latter looks more interesting, but we first need to identify all actions that can result in a Capirca output change.
Comment Actions
Change #1218739 merged by jenkins-bot:
[operations/software/homer@master] Capirca: look for the most recent completed run
Comment Actions
Change #1218209 merged by jenkins-bot:
[operations/software/netbox-extras@master] Capirca: only show diff when running in "non-commit"
Comment Actions
Thanks to the latest patches, it's now possible to see if there are pending changes to be committed to the Capirca file. Just run the script without the "commit" flag.
If there are no changes (or if it's the very first run) it will output the same file with status "completed" if there are pending changes it will "fail" while outputting the diff.
If there is a diff, then you can review it, check in the Homer policies if that host group is used somewhere (and thus on which devices to run Homer), then re-run the script with the commit flag on, then if needed, run Homer.
The failed status output doesn't impact Homer, as Homer now looks for the latest completed run.
Comment Actions
On Netbox-next:
I created an "event-rule" - https://netbox-next.wikimedia.org/extras/event-rules/1/
Configured to run the capirca script each time an IPAM | IP address object is created/updated/deleted with a few conditions to have it run a bit less often:
- status must be active
- it must be assigned to an interface
- it must have a dns_name
The first two are what the Capirca script iterate on, but the script goes a bit further by checking that the IP is assigned to a server. Unfortunately we can't do that with with "event-rule".
The 3rd one seems like a decent compromise to not have it run too often (in theory all the IPs in capirca should also be in DNS.
The main downside is if we update the IP object to a "non-active" status or detach it from an interface, etc it won't run as "event-rule" look at the "new" object's state.
As this is uncommon on a server's life-cycle, using "event-rule" seems like a decent tradeoff.
If it's causing issue we could replace (or complement) it with a regular cron job.
@cmooney If no objections I'm going to push this to prod next week.
Comment Actions
That's a great approach very nice! Haven't looked at the event-rule config but overall the idea is good with me +1
Comment Actions
event-rule enabled in prod: https://netbox.wikimedia.org/extras/event-rules/1/
Added a few lines about it in the doc: https://wikitech.wikimedia.org/w/index.php?title=Netbox&diff=2438728&oldid=2438602
Closing this task, but we can re-open if we notice any issue, or if the Homer diffs are too frequent.