Page MenuHomePhabricator

Upgrade pynetbox
Open, MediumPublic

Description

We pinned pynetbox's version in multiple locations as newer versions had breaking changes.

However we will have to upgrade at least to be able to use it with the new Netbox objects custom fields. As of today upstream is at 6.6.2:
https://github.com/netbox-community/pynetbox/releases

We're also maintaining our own deb package (5.3.0), which is used in our infra:
https://github.com/wikimedia/puppet/blob/production/modules/netbox/manifests/init.pp#L76
https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/netbox.pp#L132
But it's now in bullseye - https://packages.debian.org/bullseye/python3-pynetbox (5.3.1)
So if 5.3.1 is enough, maybe it's worth dropping our own maintained version for simplicity. It's also the last version before 6.

Updated plan:

Event Timeline

ayounsi triaged this task as Medium priority.Jun 15 2022, 8:23 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 806908 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/netbox-extras@master] Delete requirements.txt

https://gerrit.wikimedia.org/r/806908

FYI, I manually installed pynetbox 5.3.1 on netbox-dev's Netbox venv and APT package.

Change 806908 merged by Ayounsi:

[operations/software/netbox-extras@master] Delete requirements.txt

https://gerrit.wikimedia.org/r/806908

Change 820778 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/homer@master] Bump pynetbox to ~= 6.6

https://gerrit.wikimedia.org/r/820778

I manually upgraded pynetbox to 6.6.2 in netbox-dev2002:/srv/deployment/netbox .venv but I'm actually not sure something uses it. As scripts (like dns_snippet or Icinga checks) seem to use the system wide packages.

Good news is that pynetbox 6.6.0 is in sid/bookworm - https://packages.debian.org/search?searchon=names&keywords=pynetbox

generate_dns_snippets.py works fine with pynetbox 6.6.2.

2022-08-05 18:55:05,373 [INFO] Gathering devices, interfaces, addresses and prefixes from Netbox
2022-08-05 18:58:36,917 [ERROR] Unsupported assigned object type: ipam.fhrpgroup
2022-08-05 18:58:36,946 [INFO] Gathered 2679 devices from Netbox
2022-08-05 18:58:36,946 [INFO] Generating DNS records
2022-08-05 18:58:54,099 [INFO] Generated 16100 direct and reverse records (8050 each) in 30 direct zones and 243 reverse zones

The error is unrelated (see T311218#8134207)

Change 820806 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/spicerack@master] Bump pynetbox to ~= 6.6

https://gerrit.wikimedia.org/r/820806

Change 820808 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/netbox-deploy@master] Bump pynetbox to ~= 6.6

https://gerrit.wikimedia.org/r/820808

Change 820808 merged by Ayounsi:

[operations/software/netbox-deploy@master] Bump pynetbox to ~= 6.6

https://gerrit.wikimedia.org/r/820808

About Spicerack, the new sre.network.debug cookbook doesn't work with the current Spicerack pynetbox, but works on 6.6.0 (or above):

DRY-RUN: Exception raised while executing cookbook sre.network.debug:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
    raw_ret = runner.run()
  File "/srv/deployment/spicerack/cookbooks/sre/network/debug.py", line 54, in run
    self.debug_circuit(self.args.netbox_id)
  File "/srv/deployment/spicerack/cookbooks/sre/network/debug.py", line 102, in debug_circuit
    nb_circuit = self.netbox.api.circuits.circuits.get(id=netbox_id)
  File "/usr/lib/python3/dist-packages/pynetbox/core/endpoint.py", line 138, in get
    filter_lookup = self.filter(**kwargs)
  File "/usr/lib/python3/dist-packages/pynetbox/core/endpoint.py", line 216, in filter
    raise ValueError(
ValueError: A reserved ('id', 'pk', 'limit', 'offset') kwarg was passed. Please remove it try again.

I think the change was introduced in this commit https://github.com/netbox-community/pynetbox/commit/d06ca91619cf797d5e456a7836f47e132355f8dd

Change 820778 merged by jenkins-bot:

[operations/software/homer@master] Bump pynetbox to ~= 6.6

https://gerrit.wikimedia.org/r/820778

Mentioned in SAL (#wikimedia-operations) [2022-09-05T10:13:33Z] <XioNoX> upgrade python-pynetbox to 6.6 on netbox frontends - T310745

ayounsi updated the task description. (Show Details)

Change 820806 merged by jenkins-bot:

[operations/software/spicerack@master] Bump pynetbox to ~= 6.6

https://gerrit.wikimedia.org/r/820806