Page MenuHomePhabricator

Puppet: get row/rack info from Netbox
Open, NormalPublic

Description

It was discussed in the SRE-Foundations weekly meeting the need of integrating datacenter row/rack information into Puppet reading it from Netbox and exposing it in ways so that Grafana dashboards and Cumin could use it.

The requirement is to have a system that doesn't directly depend on Netbox, hence that if Netbox is down should continue to work just with stale data.

The various options discussed boils down to:

  1. Have a script that periodically generates a hiera file with that information for each host, to be merged by the puppetmasters with the public tree as we already do with the private repo
  2. Have a hiera backend that dynamically query Netbox (or a local file to have a cache)
  3. Have a Puppet function that query Netbox (TBD how to make a local cache)
  4. Use client side facts (require each host to contact Netbox with a token and also have a local cache)
  5. See if server side facts could be used for this (at first sight it seems not, but need more investigation)

As for the exposure to Cumin, it might not be needed as we've already in CR a Netbox backend for Cumin. Or could be exposed using the same trick used for $::site in profile::cumin::target.
As for the exposure to Grafana dashboards it needs investigation.

Event Timeline

Volans created this task.Jul 31 2019, 11:19 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 31 2019, 11:19 AM
Volans triaged this task as Normal priority.Jul 31 2019, 11:40 AM

It seems to me that the simplest option would be #1, it would also be the one that optimizes API calls to Netbox (just one per puppetmaster every X minutes) and has a natural caching mechanism.
In addition we could add an alert if the file is stale (too old).

Change 526664 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] netbox/puppet: An example of how we may intergrate netbox data with puppet

https://gerrit.wikimedia.org/r/526664

jbond added a comment.Jul 31 2019, 1:12 PM

Have a script that periodically generates a hiera file with that information for each host, to be merged by the puppetmasters with the public tree as we already do with the private repo

The use of the word "merged" suggests [to me] that some git operation is taking place, unless im missing something i dont think this is the case. I would suggest

Have a script that periodically generates a hiera file with that information for each host and place that file in to the appropriate place in the hiera hierarchy.

If we go this route and getting a bit more into the details i think it would probably makes more sense to create one file per host then were ever the data is needed in puppet one can just call lookup('netbox::rack'). or is there a need for some systems to have all the netbox data?

I created a quick example of how the one file per host configuration may look

If we go on the one file per host approach then I'd say we can read the file before writing so that we write/overwrite only if it's not there or has the wrong info. This should limit the re-write operations that in turn should reduce the race conditions of puppet non finding the file at the exact moment it's reading them to a negligible amount.
FYI row/rack don't change for most hosts during their lifetime, but in some cases we move hosts around, so it's a use case to take into account.