Page MenuHomePhabricator
Create Task
Maniphest T311486

Pynetbox: enable threading
Closed, ResolvedPublic
Actions

Description

From https://github.com/netbox-community/pynetbox#threading

pynetbox supports threading for .all() and .filter()

I gave it a try on netbox-dev with generate_dns_snippets.py as it contains a lot of .all() and .filter() and the performance improvement is there: 2m9s vs. 1m26s

See output on

P30535 pynetbox threading
1ayounsi@netbox-dev2002:~$ time /srv/deployment/netbox-extras/dns/generate_dns_snippets.py -v commit --icinga-check "icinga-check"
22022-06-28 08:35:42,798 [ERROR] /srv/deployment/netbox-extras/dns/generate_dns_snippets.py:655 Failed to read Icinga state from /var/run/netbox_generate_dns_snippets.state: [Errno 2] No such file or directory: '/var/run/netbox_generate_dns_snippets.state'
32022-06-28 08:35:42,799 [INFO] /srv/deployment/netbox-extras/dns/generate_dns_snippets.py:148 Gathering devices, interfaces, addresses and prefixes from Netbox
42022-06-28 08:35:42,801 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:973 Starting new HTTPS connection (1): netbox-next.wikimedia.org:443
52022-06-28 08:35:43,067 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active HTTP/1.1" 200 29777
62022-06-28 08:35:47,574 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&limit=7439&offset=50 HTTP/1.1" 200 814970
72022-06-28 08:35:52,473 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?limit=1000&offset=1050&status=active HTTP/1.1" 200 845440
82022-06-28 08:35:56,915 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?limit=1000&offset=2050&status=active HTTP/1.1" 200 831506
92022-06-28 08:36:01,441 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?limit=1000&offset=3050&status=active HTTP/1.1" 200 836596
102022-06-28 08:36:06,021 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?limit=1000&offset=4050&status=active HTTP/1.1" 200 830870
112022-06-28 08:36:11,334 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?limit=1000&offset=5050&status=active HTTP/1.1" 200 854477
122022-06-28 08:36:16,047 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?limit=1000&offset=6050&status=active HTTP/1.1" 200 857400
132022-06-28 08:36:18,009 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?limit=1000&offset=7050&status=active HTTP/1.1" 200 333294
142022-06-28 08:36:19,789 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/ HTTP/1.1" 200 59475
152022-06-28 08:36:24,235 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=15946&offset=50 HTTP/1.1" 200 1333793
162022-06-28 08:36:30,933 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=1050 HTTP/1.1" 200 1630348
172022-06-28 08:36:37,277 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=2050 HTTP/1.1" 200 1620694
182022-06-28 08:36:42,722 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=3050 HTTP/1.1" 200 1467671
192022-06-28 08:36:48,038 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=4050 HTTP/1.1" 200 1389995
202022-06-28 08:36:51,761 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=5050 HTTP/1.1" 200 1223879
212022-06-28 08:36:55,678 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=6050 HTTP/1.1" 200 1196143
222022-06-28 08:36:59,486 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=7050 HTTP/1.1" 200 1176453
232022-06-28 08:37:03,788 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=8050 HTTP/1.1" 200 1251809
242022-06-28 08:37:08,076 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=9050 HTTP/1.1" 200 1231642
252022-06-28 08:37:10,585 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=10050 HTTP/1.1" 200 1044202
262022-06-28 08:37:12,936 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=11050 HTTP/1.1" 200 1044483
272022-06-28 08:37:15,447 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=12050 HTTP/1.1" 200 1046171
282022-06-28 08:37:18,950 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=13050 HTTP/1.1" 200 1171196
292022-06-28 08:37:22,712 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=14050 HTTP/1.1" 200 1198189
302022-06-28 08:37:26,358 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=1000&offset=15050 HTTP/1.1" 200 1060690
312022-06-28 08:37:30,023 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/virtualization/interfaces/ HTTP/1.1" 200 29166
322022-06-28 08:37:30,445 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/virtualization/interfaces/?limit=261&offset=50 HTTP/1.1" 200 122764
332022-06-28 08:37:30,624 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/prefixes/ HTTP/1.1" 200 34342
342022-06-28 08:37:31,077 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/prefixes/?limit=474&offset=50 HTTP/1.1" 200 284492
352022-06-28 08:37:31,713 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/?status=active&status=planned&status=staged&status=failed&status=inventory&status=decommissioning HTTP/1.1" 200 105398
362022-06-28 08:37:34,562 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/?status=active&status=planned&status=staged&status=failed&status=inventory&status=decommissioning&limit=2421&offset=50 HTTP/1.1" 200 2022656
372022-06-28 08:37:37,049 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/?limit=1000&offset=1050&status=active&status=planned&status=staged&status=failed&status=inventory&status=decommissioning HTTP/1.1" 200 2024920
382022-06-28 08:37:39,698 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/?limit=1000&offset=2050&status=active&status=planned&status=staged&status=failed&status=inventory&status=decommissioning HTTP/1.1" 200 690082
39[...]
402022-06-28 08:37:41,278 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/virtualization/virtual-machines/ HTTP/1.1" 200 74472
412022-06-28 08:37:42,035 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/virtualization/virtual-machines/?limit=259&offset=50 HTTP/1.1" 200 310980
42
432022-06-28 08:37:42,130 [ERROR] /srv/deployment/netbox-extras/dns/generate_dns_snippets.py:173 Unsupported assigned object type: ipam.fhrpgroup
44[...]
452022-06-28 08:37:51,817 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:973 Starting new HTTPS connection (1): netbox-next.wikimedia.org:443
462022-06-28 08:37:51,874 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/extras/object-changes/?time_after=2022-06-28+08%3A07%3A51.815910 HTTP/1.1" 200 52
47[...]
48real 2m9.411s
49user 0m14.902s
50sys 0m0.847s
51
52
53
54=====================
55
56
57
58ayounsi@netbox-dev2002:~$ time /srv/deployment/netbox-extras/dns/generate_dns_snippets.py -v commit --icinga-check "icinga-check"
592022-06-28 08:39:44,240 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:973 Starting new HTTPS connection (1): netbox-next.wikimedia.org:443
602022-06-28 08:39:50,130 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&limit=0 HTTP/1.1" 200 802360
612022-06-28 08:39:50,160 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:973 Starting new HTTPS connection (2): netbox-next.wikimedia.org:443
622022-06-28 08:39:50,164 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:973 Starting new HTTPS connection (3): netbox-next.wikimedia.org:443
632022-06-28 08:39:50,174 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:973 Starting new HTTPS connection (4): netbox-next.wikimedia.org:443
642022-06-28 08:39:54,579 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&offset=1000&limit=1000 HTTP/1.1" 200 845621
652022-06-28 08:39:55,098 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&offset=3000&limit=1000 HTTP/1.1" 200 836719
662022-06-28 08:39:58,466 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&offset=4000&limit=1000 HTTP/1.1" 200 829758
672022-06-28 08:40:00,502 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&offset=2000&limit=1000 HTTP/1.1" 200 831853
682022-06-28 08:40:02,899 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&offset=5000&limit=1000 HTTP/1.1" 200 854450
692022-06-28 08:40:04,969 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&offset=7000&limit=1000 HTTP/1.1" 200 377338
702022-06-28 08:40:05,302 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/ip-addresses/?status=active&offset=6000&limit=1000 HTTP/1.1" 200 856001
712022-06-28 08:40:12,384 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?limit=0 HTTP/1.1" 200 1301453
722022-06-28 08:40:12,541 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:272 Resetting dropped connection: netbox-next.wikimedia.org
732022-06-28 08:40:12,545 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:272 Resetting dropped connection: netbox-next.wikimedia.org
742022-06-28 08:40:12,561 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:272 Resetting dropped connection: netbox-next.wikimedia.org
752022-06-28 08:40:18,880 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=2000&limit=1000 HTTP/1.1" 200 1626900
762022-06-28 08:40:19,470 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=1000&limit=1000 HTTP/1.1" 200 1636174
772022-06-28 08:40:23,605 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=4000&limit=1000 HTTP/1.1" 200 1387417
782022-06-28 08:40:26,209 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=3000&limit=1000 HTTP/1.1" 200 1479487
792022-06-28 08:40:27,469 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=5000&limit=1000 HTTP/1.1" 200 1235212
802022-06-28 08:40:30,657 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=6000&limit=1000 HTTP/1.1" 200 1194842
812022-06-28 08:40:30,841 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=7000&limit=1000 HTTP/1.1" 200 1172883
822022-06-28 08:40:34,895 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=9000&limit=1000 HTTP/1.1" 200 1241527
832022-06-28 08:40:35,644 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=8000&limit=1000 HTTP/1.1" 200 1253120
842022-06-28 08:40:37,159 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=10000&limit=1000 HTTP/1.1" 200 1044726
852022-06-28 08:40:38,655 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=11000&limit=1000 HTTP/1.1" 200 1044441
862022-06-28 08:40:39,503 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=12000&limit=1000 HTTP/1.1" 200 1045453
872022-06-28 08:40:42,720 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=13000&limit=1000 HTTP/1.1" 200 1163827
882022-06-28 08:40:43,069 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=14000&limit=1000 HTTP/1.1" 200 1199744
892022-06-28 08:40:46,562 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/interfaces/?offset=15000&limit=1000 HTTP/1.1" 200 1119428
902022-06-28 08:40:51,734 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/virtualization/interfaces/?limit=0 HTTP/1.1" 200 151724
912022-06-28 08:40:52,269 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/ipam/prefixes/?limit=0 HTTP/1.1" 200 318652
922022-06-28 08:40:54,928 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/?status=active&status=planned&status=staged&status=failed&status=inventory&status=decommissioning&limit=0 HTTP/1.1" 200 2032172
932022-06-28 08:40:54,971 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:272 Resetting dropped connection: netbox-next.wikimedia.org
942022-06-28 08:40:56,193 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/?status=active&status=planned&status=staged&status=failed&status=inventory&status=decommissioning&offset=2000&limit=1000 HTTP/1.1" 200 791553
952022-06-28 08:40:57,897 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/?status=active&status=planned&status=staged&status=failed&status=inventory&status=decommissioning&offset=1000&limit=1000 HTTP/1.1" 200 2018945
96[...]
972022-06-28 08:40:59,889 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/virtualization/virtual-machines/?limit=0 HTTP/1.1" 200 385234[...]
982022-06-28 08:41:05,906 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:272 Resetting dropped connection: netbox-next.wikimedia.org
992022-06-28 08:41:06,129 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/2717/ HTTP/1.1" 200 2148
1002022-06-28 08:41:06,360 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/3572/ HTTP/1.1" 200 2175
1012022-06-28 08:41:06,705 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/4133/ HTTP/1.1" 200 2162
1022022-06-28 08:41:06,934 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/4125/ HTTP/1.1" 200 2203
1032022-06-28 08:41:07,273 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/2305/ HTTP/1.1" 200 2029
1042022-06-28 08:41:07,546 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/dcim/devices/4124/ HTTP/1.1" 200 2198
105[...]
1062022-06-28 08:41:10,100 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:973 Starting new HTTPS connection (1): netbox-next.wikimedia.org:443
1072022-06-28 08:41:10,194 [DEBUG] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:452 https://netbox-next.wikimedia.org:443 "GET /api/extras/object-changes/?time_after=2022-06-28+08%3A11%3A10.099049&limit=0 HTTP/1.1" 200 52
108
109real 1m26.328s
110user 0m16.770s
111sys 0m0.761s
112
113

Looking at the release notes on https://github.com/netbox-community/pynetbox/releases?q=thread&expanded=true
Even though it would works on 5.3.1 (previous target with T310745: Upgrade pynetbox) 6.1.1 fixes an issue that could be problematic.

So next steps:

Note that it's a parallel effort to {T310577} but have the advantage of being a potential easy win (big impact, low cost).

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Enable pynetbox threadingAyounsioperations/software/homermaster+4 -2
Enable pynetbox threadingAyounsioperations/software/spicerackmaster+2 -2
Enable pynetbox threading for DNS/Ganeti/Mgmt scriptsAyounsioperations/software/netbox-extrasmaster+2 -2
Enable pynetbox threading for generate_dns_snippets.pyAyounsioperations/software/netbox-extrasmaster+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT362985 Improve how we generate DNS entries from Netbox
 OpenNoneT271577 Make DNS generation faster
 ResolvedayounsiT311486 Pynetbox: enable threading
 ResolvedayounsiT310745 Upgrade pynetbox

Event Timeline

ayounsi created this task.Jun 28 2022, 9:19 AM
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptJun 28 2022, 9:19 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi added a subtask: T310745: Upgrade pynetbox.Jun 28 2022, 9:19 AM
ayounsi mentioned this in T310745: Upgrade pynetbox.
ayounsi added a parent task: T271577: Make DNS generation faster.Aug 5 2022, 8:29 PM
gerritbot added a comment.Aug 11 2022, 10:36 AM

Change 822339 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/spicerack@master] Enable pynetbox threading

https://gerrit.wikimedia.org/r/822339

gerritbot added a project: Patch-For-Review.Aug 11 2022, 10:36 AM
gerritbot added a comment.Aug 30 2022, 2:42 PM

Change 828031 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/homer@master] Enable pynetbox threading

https://gerrit.wikimedia.org/r/828031

gerritbot added a comment.Aug 30 2022, 3:12 PM

Change 828035 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/netbox-extras@master] Enable pynetbox threading for DNS/Ganeti/Mgmt scripts

https://gerrit.wikimedia.org/r/828035

gerritbot added a comment.Sep 5 2022, 10:25 AM

Change 829758 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/netbox-extras@master] Enable pynetbox threading for generate_dns_snippets.py

https://gerrit.wikimedia.org/r/829758

gerritbot added a comment.Sep 5 2022, 10:28 AM

Change 829758 merged by Ayounsi:

[operations/software/netbox-extras@master] Enable pynetbox threading for generate_dns_snippets.py

https://gerrit.wikimedia.org/r/829758

ayounsi mentioned this in rOSNE48dd5db73472: Enable pynetbox threading for generate_dns_snippets.py.Sep 5 2022, 10:28 AM
ayounsi updated the task description. (Show Details)Sep 5 2022, 10:32 AM

cumin1001:~$ time sudo cookbook -d sre.dns.netbox "test"

Before: 2m54.708s
After: 1m12.567s

gerritbot added a comment.Sep 5 2022, 1:37 PM

Change 828035 merged by jenkins-bot:

[operations/software/netbox-extras@master] Enable pynetbox threading for DNS/Ganeti/Mgmt scripts

https://gerrit.wikimedia.org/r/828035

Volans mentioned this in rOSNE2cce8d65381c: Enable pynetbox threading for DNS/Ganeti/Mgmt scripts.Sep 5 2022, 1:38 PM
gerritbot added a comment.Sep 6 2022, 8:43 AM

Change 822339 merged by jenkins-bot:

[operations/software/spicerack@master] Enable pynetbox threading

https://gerrit.wikimedia.org/r/822339

gerritbot added a comment.Sep 6 2022, 10:47 AM

Change 828031 merged by jenkins-bot:

[operations/software/homer@master] Enable pynetbox threading

https://gerrit.wikimedia.org/r/828031

ayounsi mentioned this in rOSHOa4a2f131fa5a: Enable pynetbox threading.Sep 6 2022, 11:18 AM
ayounsi closed subtask T310745: Upgrade pynetbox as Resolved.Sep 27 2022, 7:53 AM
ayounsi closed this task as Resolved.Sep 27 2022, 8:00 AM
ayounsi claimed this task.
ayounsi updated the task description. (Show Details)

Everything that can is now using threading. Thanks @Volans for releasing the new Homer and Spicerack

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits