To expose [[ https://grafana.wikimedia.org/d/ppq_8SRMk/netbox-device-statistic-breakdown?orgId=1 | Netbox models stats to Prometheus ]], we currently rely on a [[ https://github.com/wikimedia/operations-software-netbox-extras/blob/master/customscripts/getstats.py | CustomScript ]], fronted by a [[ https://github.com/wikimedia/operations-software-netbox-extras/blob/master/tools/custom_script_proxy.py | uwsgi proxy ]].
The upgrade to 3.2 showed limitation to this approach
Even though it have worked fine until recently this approach have its limits:
* Database usage - see {T311048} - mitigated
* Possibly resource usage - each call to the endpoint triggers the scripts that pulls data from the DB
* Homemade - higher maintaining cost
* Limited in data exposed
There is an existing Netbox plugin that does the same things, not actively maintained, but possibly better: https://github.com/networktocode/ntc-netbox-plugin-metrics-ext
I installed on netbox-dev:
```lang=bash
# Temporarily configured the proxies
cd /srv/deployment/netbox && . venv/bin/activate
pip install ntc-netbox-plugin-metrics-ext
# Manual fix for https://github.com/networktocode/ntc-netbox-plugin-metrics-ext/pull/22
vim /srv/deployment/netbox/venv/lib/python3.9/site-packages/netbox_metrics_ext/metrics.py
# Add `PLUGINS = ["netbox_metrics_ext"]` to
vim /etc/netbox/configuration.py
```
Which exposed the following on https://netbox-next.wikimedia.org/api/plugins/metrics-ext/app-metrics
{P29937}
It should be possible to add more models with:
`PLUGINS_CONFIG` (cf the plugin's doc).
It also exposes reports stats (eg. to replace our Icinga report alerting with Prometheus) and jobs stats, to add data points to the overall system's health.
We could deploy it in parallel to our current system to test it, and then decommission the old one if the new ones works as expected.
The only blocker so far to me is on how to cleanly "backport" the fix from https://github.com/networktocode/ntc-netbox-plugin-metrics-ext/pull/22.
Edit: as our current exporter aggregates data per site, we would have to leverage the "[[ https://github.com/networktocode/ntc-netbox-plugin-metrics-ext#add-your-own-metrics | add-your-own-metrics ]]" feature or add it upstream.