Page MenuHomePhabricator

cron-spam: /usr/local/sbin/check-cumin-aliases
Closed, ResolvedPublic

Description

We are getting the following spam messages from sarin and neodymium. This is caused because the servers are unable to access the puppetdb servers

Cron Daemon root@neodymium.eqiad.wmnet via wikimedia.org 
10:48 AM (2 hours ago)
to root

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 724, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 203, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 133, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 87, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 78, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 559, in urlopen
    _pool=self, _stacktrace=stacktrace)
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 658, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 724, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 203, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 133, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 87, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 78, in create_connection
    sock.connect(sa)
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/sbin/check-cumin-aliases", line 73, in <module>
    sys.exit(main())
  File "/usr/local/sbin/check-cumin-aliases", line 35, in main
    all_hosts = query.Query(config).execute('*')
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 59, in execute
    hosts = self._query_default_backend(query_string)
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 94, in _query_default_backend
    return query.execute(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 47, in execute
    return self._execute()
  File "/usr/lib/python3/dist-packages/cumin/backends/puppetdb.py", line 254, in _execute
    hosts = self._api_call(query)
  File "/usr/lib/python3/dist-packages/cumin/backends/puppetdb.py", line 505, in _api_call
    resources = requests.post(self.url + self.endpoint, json={'query': query}, verify=True)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 94, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 407, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))

Event Timeline

jbond triaged this task as Medium priority.May 3 2019, 11:50 AM
jbond created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 3 2019, 11:50 AM

Change 507948 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] cumin: update list of cumin_masters

https://gerrit.wikimedia.org/r/507948

Change 507948 abandoned by Jbond:
cumin: update list of cumin_masters

Reason:
theses servers are due for decommision

https://gerrit.wikimedia.org/r/507948

jbond closed this task as Resolved.Jun 19 2019, 2:29 PM

Theses boxes where switch to spare so spam should be halted please reopen if not