Page MenuHomePhabricator
Create Task
Maniphest T358339

Netbox: capirca.getHosts script runs into timeout
Closed, DuplicatePublic
Actions

Description

I tried to remove cumin1001 from the router ACLs. The same happened before when adding a Puppet server to the router ACLs, then script runs, but eventually reaches the 5 minutes timeout when running https://netbox.wikimedia.org/extras/scripts/capirca.GetHosts/:

An exception occurred: JobTimeoutException: Task exceeded maximum timeout value (300 seconds)

Traceback (most recent call last):
  File "/srv/deployment/netbox/current/src/netbox/extras/scripts.py", line 461, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/srv/deployment/netbox-extras//customscripts/capirca.py", line 75, in run
    hostname, ip = self.process_ipaddress(ipaddress)
  File "/srv/deployment/netbox-extras//customscripts/capirca.py", line 27, in process_ipaddress
    if ipaddress.assigned_object_type == self.interface_ct and ipaddress.assigned_object:
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/contrib/contenttypes/fields.py", line 250, in __get__
    rel_obj = ct.get_object_for_this_type(pk=pk_val)
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/contrib/contenttypes/models.py", line 173, in get_object_for_this_type
    return self.model_class()._base_manager.using(self._state.db).get(**kwargs)
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/models/query.py", line 492, in get
    num = len(clone)
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/models/query.py", line 302, in __len__
    self._fetch_all()
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/models/query.py", line 1507, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1361, in execute_sql
    cursor.execute(sql, params)
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django_prometheus/db/common.py", line 71, in execute
    return super().execute(*args, **kwargs)
  File "/usr/lib/python3.9/encodings/utf_8.py", line 15, in decode
    def decode(input, errors='strict'):
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/rq/timeouts.py", line 63, in handle_death_penalty
    raise self._exception('Task exceeded maximum timeout value ' '({0} seconds)'.format(self._timeout))
rq.timeouts.JobTimeoutException: Task exceeded maximum timeout value (300 seconds)

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.Feb 23 2024, 1:43 PM
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptFeb 23 2024, 1:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi added a comment.Feb 23 2024, 1:52 PM

https://netbox.wikimedia.org/admin/extras/jobresult/?name=capirca.GetHosts&o=-3.1.2
Not a good track record.

Screenshot 2024-02-23 at 14-50-28 Select job result to change NetBox.png (1,549×664 px, 119 KB)

We can bump the timeout, but the script shouldn't even need 5min.

Seems similar to T341843: Netbox rq.timeouts.JobTimeoutException.

ayounsi triaged this task as High priority.Feb 23 2024, 1:53 PM
ayounsi added a subtask: T336275: Upgrade Netbox to 4.x.Feb 23 2024, 2:05 PM
ayounsi mentioned this in T361549: Automatically run Capirca Netbox script regularly.Apr 2 2024, 7:47 AM
ayounsi added a parent task: T361549: Automatically run Capirca Netbox script regularly.Apr 8 2024, 2:28 PM
ayounsi added a comment.EditedAug 1 2024, 10:33 AM

Still happening in Netbox 4 (https://netbox.wikimedia.org/core/jobs/56219/):

Traceback (most recent call last):
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py\", line 236, in __get__
    rel_obj = self.field.get_cached_value(instance)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/fields/mixins.py\", line 15, in get_cached_value
    return instance._state.fields_cache[cache_name]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'role'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/srv/deployment/netbox/current/src/netbox/extras/scripts.py\", line 662, in _run_script
    script.output = script.run(data, commit)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/netbox/customscripts/capirca.py\", line 79, in run
    hostname, ip = self.process_ipaddress(ipaddress)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/netbox/customscripts/capirca.py\", line 38, in process_ipaddress
    if not ipaddress.assigned_object.device.role.slug == \"server\":
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py\", line 254, in __get__
    rel_obj = self.get_object(instance)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py\", line 217, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py\", line 645, in get
    num = len(clone)
          ^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py\", line 382, in __len__
    self._fetch_all()
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py\", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py\", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py\", line 1562, in execute_sql
    cursor.execute(sql, params)
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py\", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py\", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py\", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/django_prometheus/db/common.py\", line 69, in execute
    return super().execute(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/psycopg/cursor.py\", line 93, in execute
    self._conn.wait(
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py\", line 394, in wait
    return waiting.wait(gen, self.pgconn.socket, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"psycopg_c/_psycopg/waiting.pyx\", line 201, in psycopg_c._psycopg.wait_c
  File \"/srv/deployment/netbox/venv/lib/python3.11/site-packages/rq/timeouts.py\", line 63, in handle_death_penalty
    raise self._exception('Task exceeded maximum timeout value ({0} seconds)'.format(self._timeout))
rq.timeouts.JobTimeoutException: Task exceeded maximum timeout value (900 seconds)
ayounsi closed this task as a duplicate of T341843: Netbox rq.timeouts.JobTimeoutException.Aug 6 2024, 12:58 PM
ayounsi closed subtask T336275: Upgrade Netbox to 4.x as Resolved.Aug 7 2024, 9:37 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits