Page MenuHomePhabricator
Create Task
Maniphest T243927

Export Netbox Stats for DCops to a visualization tool
Closed, ResolvedPublic
Actions

Description

Papaul has requested an automated system to export statistics from Netbox into (prometheus/graphana|kibana).

After some discussion with observability, there are some possible solutions:

  1. Export aggregate data (counts of server type, manufacturer, status) tagged with data center to Prometheus via a mini web applet that queries the Netbox API and exposes a prometheus data set.
  1. Create an SQL query based dashboard in Graphana to query the Netbox database directly.

Primarily the interest is in counts of model number and status broken down by data center so graphs of these is the main deliverable.

Details

SubjectRepoBranchLines +/-
netbox scriptproxy: Add listen stanza for 8443 for the scriptproxyoperations/puppetproduction+3 -0
prometheus::ops: Add prometheus job to scrape Netbox scriptsoperations/puppetproduction+20 -1
netbox: Add framework for exposing scripts to internal servicesoperations/puppetproduction+101 -0
Add support for getting Device status breakdownsoperations/software/netbox-extrasmaster+74 -0
Customize query in gerrit

Related Objects

Event Timeline

• crusnov created this task.Jan 29 2020, 10:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 29 2020, 10:55 PM
Papaul added a subscriber: wiki_willy.Jan 29 2020, 11:02 PM
• crusnov added projects: DC-Ops, netbox, SRE-tools.Jan 29 2020, 11:11 PM
• crusnov moved this task from Ready to In Progress on the User-crusnov board.Feb 4 2020, 9:28 PM
gerritbot added a comment.Feb 24 2020, 11:15 PM

Change 574600 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/software/netbox-extras@master] Add support for getting Device status breakdowns

https://gerrit.wikimedia.org/r/574600

gerritbot added a project: Patch-For-Review.Feb 24 2020, 11:15 PM
gerritbot added a comment.Feb 25 2020, 9:11 PM

Change 574600 merged by CRusnov:
[operations/software/netbox-extras@master] Add support for getting Device status breakdowns

https://gerrit.wikimedia.org/r/574600

Maintenance_bot removed a project: Patch-For-Review.Feb 25 2020, 10:11 PM
gerritbot added a comment.Feb 28 2020, 7:49 PM

Change 575603 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] netbox: Add framework for exposing scripts to internal services

https://gerrit.wikimedia.org/r/575603

gerritbot added a project: Patch-For-Review.Feb 28 2020, 7:49 PM
gerritbot added a comment.Mar 3 2020, 10:29 PM

Change 576459 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] prometheus::ops: Add prometheus job to scrape Netbox scripts

https://gerrit.wikimedia.org/r/576459

• crusnov moved this task from Backlog to Patches / Testing / Pending on the netbox board.Apr 14 2020, 5:35 PM
• crusnov added a comment.Apr 14 2020, 5:42 PM

This has patches pending final review. Should push to production soon.

• crusnov triaged this task as Medium priority.Apr 15 2020, 3:32 PM
gerritbot added a comment.Apr 28 2020, 5:21 PM

Change 575603 merged by CRusnov:
[operations/puppet@production] netbox: Add framework for exposing scripts to internal services

https://gerrit.wikimedia.org/r/575603

• crusnov moved this task from In Progress to Pending on the User-crusnov board.May 1 2020, 9:58 PM
gerritbot added a comment.May 12 2020, 4:21 PM

Change 576459 merged by CRusnov:
[operations/puppet@production] prometheus::ops: Add prometheus job to scrape Netbox scripts

https://gerrit.wikimedia.org/r/576459

Maintenance_bot removed a project: Patch-For-Review.May 12 2020, 5:10 PM
ayounsi subscribed.May 18 2020, 8:01 AM

FYI, Prometheus is trying to query netbox2001.wikimedia.org:8443 but there is nothing listening on that port. Which is causing this alert:
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=icinga1001&service=Prometheus+jobs+reduced+availability

Prometheus jobs reduced availability - job=netbox_device_statistics site=codfw

As it's a generic alert I only ACKed it for 2 days.

ayounsi raised the priority of this task from Medium to High.May 21 2020, 10:03 AM
• crusnov added a comment.May 21 2020, 4:17 PM

Thanks for the ping, I will get to it today.

gerritbot added a comment.May 21 2020, 7:01 PM

Change 597851 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] netbox scriptproxy: Add listen stanza for 8443 for the scriptproxy

https://gerrit.wikimedia.org/r/597851

gerritbot added a project: Patch-For-Review.May 21 2020, 7:01 PM
gerritbot added a comment.May 21 2020, 8:45 PM

Change 597851 merged by CRusnov:
[operations/puppet@production] netbox scriptproxy: Add listen stanza for 8443 for the scriptproxy

https://gerrit.wikimedia.org/r/597851

• crusnov closed this task as Resolved.May 21 2020, 8:54 PM

Okay this should be fix

In T243927#6143794, @ayounsi wrote:

FYI, Prometheus is trying to query netbox2001.wikimedia.org:8443 but there is nothing listening on that port. Which is causing this alert:
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=icinga1001&service=Prometheus+jobs+reduced+availability

Prometheus jobs reduced availability - job=netbox_device_statistics site=codfw

As it's a generic alert I only ACKed it for 2 days.

I have fixed the issue with nb2001. This should be resolved.

Maintenance_bot removed a project: Patch-For-Review.May 21 2020, 9:10 PM
ayounsi mentioned this in T262898: Graph reports status in Prometheus.Sep 15 2020, 7:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL