Change 842244 had a related patch set uploaded (by Ayounsi; author: XioNoX):

[operations/software/netbox-deploy@3-2-2] Add ntc-netbox-plugin-metrics-ext plugin

https://gerrit.wikimedia.org/r/842244

Change 842244 merged by Ayounsi:

[operations/software/netbox-deploy@3-2-2] Add ntc-netbox-plugin-metrics-ext plugin

https://gerrit.wikimedia.org/r/842244

Change 842417 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/netbox-deploy@3-2-2] Re-generate wheels with pip list

https://gerrit.wikimedia.org/r/842417

Change 842417 merged by Ayounsi:

[operations/software/netbox-deploy@3-2-2] Re-generate wheels with pip list

https://gerrit.wikimedia.org/r/842417

Change 842433 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/netbox-deploy@3-2-2] Netbox: fix TZdata bug

https://gerrit.wikimedia.org/r/842433

Change 842433 merged by Ayounsi:

[operations/software/netbox-deploy@3-2-2] Netbox: fix TZdata bug

https://gerrit.wikimedia.org/r/842433

After many iterations (see patches above) the pypi version ended up being deployed in prod...

I manually applied the fix to venv/lib/python3.9/site-packages/netbox_metrics_ext/metrics.py as a temporary test and enabled the plugin in /etc/netbox/configuration.py

The result on http://netbox.wikimedia.org/api/plugins/metrics-ext/app-metrics is a 695911 lines 65M file, that takes 17s to generate. Most of it being the reports (41165 job results on https://netbox.wikimedia.org/admin/extras/jobresult/x 4 status x multiple checks).
See the output attached bellow.

It would still be valuable to have the plugin running with "reports": False but we can probably also trim the reports history (and maybe reduce retention from the current 90 days). Especially as this endpoint would be exposed to the world.

nice work and sounds good to me

The only blocker so far to me is on how to cleanly "backport" the fix from https://github.com/networktocode/ntc-netbox-plugin-metrics-ext/pull/22.

Im not sure what the policy is about forking to the wikimedia github organisation but if that is not contentious we could fork it there

Edit: as our current exporter aggregates data per site, we would have to leverage the "add-your-own-metrics" feature or add it upstream.

Its probably better to add this as a feture and potentially overload the reports config option to have reports: Union[bool, Enum['aggregated']]

It would still be valuable to have the plugin running with "reports": False

See above if we create a feature for aggregated reports its a double win :)

There is also https://github.com/TheDJVG/netbox-more-metrics

I'm not really loving either of the metrics plugins. The ntc-netbox-plugin-metrics-ext plugins seems like the author either didn't know how to extend the existing metrics endpoint or wanted something rather specific. netbox-more-metrics currently have a bug, that prevents it from running on the newest version of Netbox, it rely on some models that have changed.

The ntc-netbox plugin might be the easiest to fix up, but it still generates to much data.

There's also the option to simply extend our getstats.py and modify it to dump it's data into the Node Exporter node.d directory and regular intervals. We don't really need minute to minute statistics from Netbox. We simply modify the getstats to be a management command and run it on a timer once an hour or so. Looking at the existing code there's a should be a few Django ORM tricks we can employ to speed things up as well.

The main objection to using our own plugin is that it will be higher maintance cost, but from the history of the other plugins, it might be a cost we need to take on anyway and in that case the netbox-more-metrics while flexible might be a bit much.

The getstats netbox script is a hack and we should at best replace it with a Netbox plugin. netbox-more-metrics also seems more flexible to export data about more objects.

Not sure how the node exporter approach would work in a active/active setup. There we would want Django metrics from both instances, but the alerts/objects stats only once.

for information https://github.com/Eskemm-Numerique/ntc-netbox-plugin-metrics-ext/pull/1 got merged, so ntc-netbox-plugin-metrics-ext should now works out of the box.