Change 464081 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[labs/private@master] Add fake ssh keys for netbox user

https://gerrit.wikimedia.org/r/464081

Change 464082 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Netbox, set the napalm_username variable and matching keyholder

https://gerrit.wikimedia.org/r/464082

Change 464081 merged by Ayounsi:
[labs/private@master] Add fake ssh keys for netbox user

https://gerrit.wikimedia.org/r/464081

Change 464082 merged by Ayounsi:
[operations/puppet@production] Netbox, set the napalm_username variable and matching keyholder

https://gerrit.wikimedia.org/r/464082

https://netbox.wikimedia.org/api/dcim/devices/1945/napalm/

NAPALM is not installed. Please see the documentation for instructions.

While

netmon1002:/srv/deployment/netbox/venv# . bin/activate
(venv) root@netmon1002:/srv/deployment/netbox/venv# python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import napalm
>>> print(napalm.__version__)
2.3.2

I checked the current code and I see that in the same try/except that raises that error it does also:

from napalm_base.exceptions import ConnectAuthError, ModuleImportError

and napalm_base is not in the virtualenv.
I've also noticed that this specific code has changed recently. If you're not too in a hurry I think we could just see if it works after upgrading Netbox, thing that I was planning to do early next week.

The upgrade solved the issue, and brought the next one.
@Volans investigated it and it seems like we're hitting that one: https://github.com/paramiko/paramiko/issues/23
In short, we have netbox and rancid ssh keys in keyholder, and NAPALM is only able to use the netbox key when it's the first one exposed by the agent.
(this is because NAPALM uses junos-pyez, which uses netmiko, which uses paramiko).

We got that one device working for testing: https://netbox.wikimedia.org/dcim/devices/1945/ by reloading keyholder (which probably shuffled the keys in the good order), but the usefulness of NAPALM in Netbox is so far limited to:

• LLDP neighbor
• some devices facts
• configuration

The first two are already being pulled by LibreNMS and the last one by rancid (and displayed in LibreNMS), and it doesn't seem like Netbox is doing any extra "processing" on LLDP neighbors (eg. checking coherency).

So enabling NAPALM in Netbox might not be worth the hassle of solving the issue above and adding an extra ssh key to all the network devices.

In addition Faidon pointed me to https://github.com/digitalocean/netbox/blob/develop/netbox/extras/management/commands/run_inventory.py

Which could be useful to populate linecards and other inventory items with their serial#, especially the ones we have support contract on.
It uses paramiko/NCclient directly probably because this is not supported by Napalm, which also mean it requires a user with a password on the network device.
there is also the risk of the data becoming outdated, as the script needs to be ran manually. On the plus side, most of the component should not be updated regularly.
Another unknown is how this handles virtual chassis, as it would probably set everything under the master member.

If we have an immediate need for that data (eg. reporting), LibreNMS already pulls all the components via SNMP (eg. https://librenms.wikimedia.org/device/device=88/tab=entphysical/).

So tl;dr, my take on it is 1/ it could be useful down the road, especially when we get closer to Netbox being the only source of truth, either by using that script on adding it to the current import script being worked on. And 2/ it's outside the scope of that task.

So the aforementioned functionality was removed as obsolete due to NAPALM support replacing it and will not be part of the 2.5 release. The inventory data models remain in the tree AIUI, and one could write external scripts to populate those, that would either use SNMP or ncclient with public key auth etc. to fetch this information. I think it would be interesting to explore, and indeed, probably more interesting than NAPALM itself.

Regarding this task's objective around NAPALM: the paramiko/netmiko keyholder issue mentioned above would probably be implicitly addressed (or rather, worked around) with the recent and undeployed keyholder rewrite and specifically I8f2aa39de58cc25da104b2f92f541035cbccf32e. That said, I'm not sure if it's worth our time to explore further in general. LLDP neighbor and config is not all that useful, but I'm wondering what "some devices facts" means exactly -- @ayounsi?

Regardless, I think this all boils down to these two questions:

• Is it worth our time/effort to pursue this NAPALM exploration further? Having NAPALM in our toolbox may or may not be interesting for netops by itself, so we should factor that in in our decision.
• If not, should we pivot this part of our goal to "inventory population in Netbox" or is this a broader and bigger task that should be followed up on at a later stage?

Thoughts by everyone and especially @ayounsi @Volans and @crusnov would be very much welcome :)

@faidon

In T205898#4735131, @faidon wrote:

Regardless, I think this all boils down to these two questions:

• Is it worth our time/effort to pursue this NAPALM exploration further? Having NAPALM in our toolbox may or may not be interesting for netops by itself, so we should factor that in in our decision.

I cannot reply for the netops part, but for the Netbox part it seems to me that the NAPALM integration is not actually useful so we could remove it from Netbox.

• If not, should we pivot this part of our goal to "inventory population in Netbox" or is this a broader and bigger task that should be followed up on at a later stage?

This is an interesting idea, I think we could investigate that a bit and have ideas on how to use it. As of now Netbox allows to keep those just as inventory items, without additional features. So I think that instead of being dragged by the inventory fever of tracking everything just because we can, we should evaluate first how we will use this data once stored in Netbox. And based on which useful use case we found act accordingly.
I'm ok to pivot this goal part to "investigate possible usage of Inventory items in Netbox" more than populating blindly items without a use case.

In T205898#4735131, @faidon wrote:

[...] but I'm wondering what "some devices facts" means exactly

Uptime, OS version, serial#, etc...
And it only displays it as far as I know.

In T205898#4735131, @faidon wrote:

• Is it worth our time/effort to pursue this NAPALM exploration further? Having NAPALM in our toolbox may or may not be interesting for netops by itself, so we should factor that in in our decision.

I don't think it's worth it for now, NAPALM with Netbox doesn't bring much/any value.
If we start using NAPALM more down the road to actually configure devices, we would use our users, (or a automation userwith right access), and most likely a different architecture thought from the ground up, and not based on Netbox's NAPALM.

I think we have consensus on the NAPALM stuff :)

In T205898#4758699, @Volans wrote:

This is an interesting idea, I think we could investigate that a bit and have ideas on how to use it. As of now Netbox allows to keep those just as inventory items, without additional features. So I think that instead of being dragged by the inventory fever of tracking everything just because we can, we should evaluate first how we will use this data once stored in Netbox. And based on which useful use case we found act accordingly.
I'm ok to pivot this goal part to "investigate possible usage of Inventory items in Netbox" more than populating blindly items without a use case.

That's a fair point! For Juniper, some of the HW (e.g. linecards) is expensive stuff -often more than entire routers or servers- and are listed separately in support contract quotes etc. I think @RobH is tracking and cross-checking that information manually at the moment (and doing so every year), so I think there is room for improvement there. I can think of more ideas and use cases but I'll take the inventory fever feedback and stop here for now :)

Mentioned in SAL (#wikimedia-operations) [2018-11-29T17:55:45Z] <XioNoX> remove test netbox user from cr3-ulsfo - T205898

Change 476581 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Revert "Netbox, set the napalm_username variable and matching keyholder"

https://gerrit.wikimedia.org/r/476581

Change 476581 abandoned by Ayounsi:
Revert "Netbox, set the napalm_username variable and matching keyholder"

Reason:
Merge conflict

https://gerrit.wikimedia.org/r/476581

Change 476583 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Revert "Netbox, set the napalm_username variable and matching keyholder"

https://gerrit.wikimedia.org/r/476583

Change 476584 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[labs/private@master] Revert "Add fake ssh keys for netbox user"

https://gerrit.wikimedia.org/r/476584

Change 476583 merged by Ayounsi:
[operations/puppet@production] Revert "Netbox, set the napalm_username variable and matching keyholder"

https://gerrit.wikimedia.org/r/476583

Change 476584 merged by Ayounsi:
[labs/private@master] Revert "Add fake ssh keys for netbox user"

https://gerrit.wikimedia.org/r/476584

All test configuration for Netbox/Napalm has been removed.