Page MenuHomePhabricator
Create Task
Maniphest T397003

Configure a prometheus dead man's snitch alert
Closed, ResolvedPublic
Actions

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
metamonitoring: add icinga moduleoperations/puppetproduction+201 -18
metamonitoring/icinga/ext-mon: add dummy basic auth infolabs/privatemaster+1 -1
metamonitoring/icinga/ext-mon: add dummy basic auth infolabs/privatemaster+4 -0
metamonitoring: avoid unnecessary public endpoint restartsoperations/puppetproduction+2 -2
metamonitoring: cleanup unneeded env filesoperations/puppetproduction+0 -21
metamonitoring: add env vars to uwsgi processoperations/puppetproduction+13 -4
metamonitoring: replace deadmanswitchamhook's Gunicorn with uWSGIoperations/puppetproduction+17 -69
metamonitoring: prepare deadmanswitchamhook's Gunicorn replacement with uWSGIoperations/puppetproduction+39 -16
metamonitoring: rename uwsgi resourceoperations/puppetproduction+5 -5
metamonitoring: cleanup gunicorn related resourcesoperations/puppetproduction+7 -47
metamonitoring: replace Gunicorn with uWSGIoperations/puppetproduction+40 -5
prom/metamon: add a dedicated sysuser for the daemonsoperations/puppetproduction+31 -25
prom/metamonitor: force gunicorn to log to a fileoperations/puppetproduction+37 -4
prom/metamonitor: add listen_port to public_endpoint vhost templateoperations/puppetproduction+5 -4
prom/metamonitor: fix indentation on karma erb config fileoperations/puppetproduction+2 -2
prom/metamonitor: fix typo on karma erb config fileoperations/puppetproduction+1 -1
prom/metamonitor: hide DeadManSwitch alerts in Karmaoperations/puppetproduction+52 -22
prom/metamonitor: simplify PQL query to retrieve instance listoperations/puppetproduction+2 -2
prometheus::pop: manage pop Prometheus instances centrallyoperations/puppetproduction+1 -1
prom/metamonitor: add CNAMEs for metamonitoring endpointsoperations/dnsmaster+4 -0
prom/metamonitor: make physical vhosts agnostic to the machine hostnameoperations/puppetproduction+19 -19
prom/metamonitor: add dead man switch and public endpointoperations/puppetproduction+598 -0
prom/metamonitor: add dead man switch and public endpointoperations/puppetproduction+568 -0
prom/metamonitor: add dead man snitch and public endpointoperations/puppetproduction+448 -1
deadmansnitch: add dms alert and am hookoperations/puppetproduction+158 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

tappof created this task.Jun 16 2025, 9:05 AM
gerritbot added a comment.Jun 16 2025, 9:06 AM

Change #1159369 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] deadmansnitch: add dms alert and am hook

https://gerrit.wikimedia.org/r/1159369

gerritbot added a project: Patch-For-Review.Jun 16 2025, 9:06 AM
gerritbot added a comment.Jul 1 2025, 1:03 PM

Change #1165514 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: add dead man snitch and public endpoint

https://gerrit.wikimedia.org/r/1165514

gerritbot added a comment.Jul 1 2025, 1:05 PM

Change #1159369 abandoned by Tiziano Fogli:

[operations/puppet@production] deadmansnitch: add dms alert and am hook

Reason:

Replaced by a new patchset.

https://gerrit.wikimedia.org/r/1159369

gerritbot added a comment.Jul 1 2025, 1:08 PM

Change #1165518 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: add dead man snitch and public endpoint

https://gerrit.wikimedia.org/r/1165518

gerritbot added a comment.Jul 1 2025, 1:10 PM

Change #1165514 abandoned by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: add dead man snitch and public endpoint

Reason:

The Change-Id was regenerated by mistake.

https://gerrit.wikimedia.org/r/1165514

gerritbot added a comment.Jul 8 2025, 9:36 AM

Change #1167157 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: add dead man switch and public endpoint

https://gerrit.wikimedia.org/r/1167157

gerritbot added a comment.Jul 8 2025, 9:37 AM

Change #1165518 abandoned by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: add dead man switch and public endpoint

Reason:

The Change-Id was regenerated by mistake.

https://gerrit.wikimedia.org/r/1165518

gerritbot added a comment.Jul 15 2025, 12:26 PM

Change #1167157 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: add dead man switch and public endpoint

https://gerrit.wikimedia.org/r/1167157

Maintenance_bot removed a project: Patch-For-Review.Jul 15 2025, 12:31 PM
gerritbot added a comment.Jul 15 2025, 2:32 PM

Change #1169668 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/dns@master] prom/metamonitor: add CNAMEs for metamonitoring endpoints

https://gerrit.wikimedia.org/r/1169668

gerritbot added a project: Patch-For-Review.Jul 15 2025, 2:32 PM
gerritbot added a comment.Jul 16 2025, 9:47 AM

Change #1170104 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: make physical vhosts agnostic to the machine hostname

https://gerrit.wikimedia.org/r/1170104

gerritbot added a comment.Jul 16 2025, 1:40 PM

Change #1170104 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: make physical vhosts agnostic to the machine hostname

https://gerrit.wikimedia.org/r/1170104

gerritbot added a comment.Jul 16 2025, 1:49 PM

Change #1169668 merged by Tiziano Fogli:

[operations/dns@master] prom/metamonitor: add CNAMEs for metamonitoring endpoints

https://gerrit.wikimedia.org/r/1169668

Maintenance_bot removed a project: Patch-For-Review.Jul 16 2025, 2:31 PM
gerritbot added a comment.Jul 17 2025, 9:46 AM

Change #1170282 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prometheus::pop: manage pop Prometheus instances centrally

https://gerrit.wikimedia.org/r/1170282

gerritbot added a project: Patch-For-Review.Jul 17 2025, 9:46 AM
gerritbot added a comment.Jul 17 2025, 9:58 AM

Change #1170282 merged by Tiziano Fogli:

[operations/puppet@production] prometheus::pop: manage pop Prometheus instances centrally

https://gerrit.wikimedia.org/r/1170282

gerritbot added a comment.Jul 17 2025, 10:20 AM

Change #1170286 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: simplify PQL query to retrieve instance list

https://gerrit.wikimedia.org/r/1170286

gerritbot added a comment.Jul 17 2025, 10:26 AM

Change #1170286 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: simplify PQL query to retrieve instance list

https://gerrit.wikimedia.org/r/1170286

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2025, 10:31 AM
gerritbot added a comment.Jul 17 2025, 2:34 PM

Change #1170360 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: hide DeadManSwitch alerts in Karma

https://gerrit.wikimedia.org/r/1170360

gerritbot added a project: Patch-For-Review.Jul 17 2025, 2:34 PM
gerritbot added a comment.Jul 18 2025, 12:24 PM

Change #1170360 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: hide DeadManSwitch alerts in Karma

https://gerrit.wikimedia.org/r/1170360

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2025, 12:31 PM
gerritbot added a comment.Jul 18 2025, 12:35 PM

Change #1170540 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: fix typo on karma erb config file

https://gerrit.wikimedia.org/r/1170540

gerritbot added a project: Patch-For-Review.Jul 18 2025, 12:35 PM

Change #1170540 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: fix typo on karma erb config file

https://gerrit.wikimedia.org/r/1170540

gerritbot added a comment.Jul 18 2025, 12:45 PM

Change #1170545 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: fix indentation on karma erb config file

https://gerrit.wikimedia.org/r/1170545

gerritbot added a comment.Jul 18 2025, 12:46 PM

Change #1170545 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: fix indentation on karma erb config file

https://gerrit.wikimedia.org/r/1170545

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2025, 1:31 PM
gerritbot added a comment.Jul 21 2025, 10:28 AM

Change #1171170 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: add listen_port to public_endpoint vhost template

https://gerrit.wikimedia.org/r/1171170

gerritbot added a project: Patch-For-Review.Jul 21 2025, 10:28 AM

Change #1171171 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamonitor: force gunicorn to log to a file

https://gerrit.wikimedia.org/r/1171171

gerritbot added a comment.Jul 21 2025, 1:55 PM

Change #1171170 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: add listen_port to public_endpoint vhost template

https://gerrit.wikimedia.org/r/1171170

gerritbot added a comment.Jul 21 2025, 1:55 PM

Change #1171171 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamonitor: force gunicorn to log to a file

https://gerrit.wikimedia.org/r/1171171

Maintenance_bot removed a project: Patch-For-Review.Jul 21 2025, 2:30 PM
gerritbot added a comment.Jul 22 2025, 9:16 AM

Change #1171546 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prom/metamon: add a dedicated sysuser for the daemons

https://gerrit.wikimedia.org/r/1171546

gerritbot added a project: Patch-For-Review.Jul 22 2025, 9:16 AM
gerritbot added a comment.Jul 22 2025, 10:41 AM

Change #1171546 merged by Tiziano Fogli:

[operations/puppet@production] prom/metamon: add a dedicated sysuser for the daemons

https://gerrit.wikimedia.org/r/1171546

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2025, 11:31 AM
tappof closed this task as Resolved.Aug 1 2025, 12:26 PM

"Dead Man Switch" alerts have been configured.
The implementation has been documented here: https://wikitech.wikimedia.org/wiki/Prometheus#Meta-Monitoring.

tappof mentioned this in T281812: Audit/Assess meta monitoring strategy.Aug 1 2025, 12:32 PM
Jelto subscribed.Oct 2 2025, 6:53 AM

Today the metamonitoring pa.ged on-call SREs. When checking the metamonitoring_public_endpoint.service on alert1002 I see a python stack trace during the time of the the alert:

Oct 02 06:09:41 alert1002 systemd[1]: Stopping metamonitoring_public_endpoint.service - Gunicorn for metamonitor_public_endpoint...
Oct 02 06:09:41 alert1002 gunicorn[3038713]: --- Logging error ---
Oct 02 06:09:41 alert1002 gunicorn[3038713]: --- Logging error ---
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Traceback (most recent call last):
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 224, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     handler()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 257, in handle_term
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     raise StopIteration
Oct 02 06:09:41 alert1002 gunicorn[3038713]: StopIteration
Oct 02 06:09:41 alert1002 gunicorn[3038713]: During handling of the above exception, another exception occurred:
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Traceback (most recent call last):
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1114, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.flush()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1094, in flush
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.stream.flush()
Oct 02 06:09:41 alert1002 gunicorn[3038713]: RuntimeError: reentrant call inside <_io.BufferedWriter name='/var/log/o11y-metamonitoring/public_endpoint.log'>
Oct 02 06:09:41 alert1002 gunicorn[3038713]: During handling of the above exception, another exception occurred:
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Traceback (most recent call last):
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1114, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.flush()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1094, in flush
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.stream.flush()
Oct 02 06:09:41 alert1002 gunicorn[3038713]: RuntimeError: reentrant call inside <_io.BufferedWriter name='/var/log/o11y-metamonitoring/public_endpoint.log'>
Oct 02 06:09:41 alert1002 gunicorn[3038713]: During handling of the above exception, another exception occurred:
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Traceback (most recent call last):
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1114, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.flush()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1094, in flush
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.stream.flush()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 242, in handle_chld
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.reap_workers()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 530, in reap_workers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.log.warning(
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/glogging.py", line 261, in warning
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.error_log.warning(msg, *args, **kwargs)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1501, in warning
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self._log(WARNING, msg, args, **kwargs)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1634, in _log
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.handle(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1644, in handle
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.callHandlers(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     hdlr.handle(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 978, in handle
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.emit(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1230, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     StreamHandler.emit(self, record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1118, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.handleError(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1031, in handleError
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     sys.stderr.write('--- Logging error ---\n')
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 242, in handle_chld
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.reap_workers()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 530, in reap_workers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.log.warning(
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/glogging.py", line 261, in warning
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.error_log.warning(msg, *args, **kwargs)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1501, in warning
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self._log(WARNING, msg, args, **kwargs)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1634, in _log
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.handle(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1644, in handle
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.callHandlers(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     hdlr.handle(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 978, in handle
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.emit(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1230, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     StreamHandler.emit(self, record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1118, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.handleError(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1031, in handleError
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     sys.stderr.write('--- Logging error ---\n')
Oct 02 06:09:41 alert1002 gunicorn[3038713]: RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Call stack:
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/bin/gunicorn", line 33, in <module>
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     sys.exit(load_entry_point('gunicorn==20.1.0', 'console_scripts', 'gunicorn')())
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 67, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/app/base.py", line 231, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     super().run()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/app/base.py", line 72, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     Arbiter(self).run()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 227, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.halt()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 342, in halt
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.stop()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 393, in stop
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     time.sleep(0.1)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 242, in handle_chld
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.reap_workers()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 513, in reap_workers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     wpid, status = os.waitpid(-1, os.WNOHANG)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 242, in handle_chld
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.reap_workers()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 530, in reap_workers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.log.warning(
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/glogging.py", line 261, in warning
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.error_log.warning(msg, *args, **kwargs)
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Message: 'Worker with pid %s was terminated due to signal %s'
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Arguments: (3039077, 15)
Oct 02 06:10:11 alert1002 systemd[1]: metamonitoring_public_endpoint.service: Deactivated successfully.
Oct 02 06:10:11 alert1002 systemd[1]: Stopped metamonitoring_public_endpoint.service - Gunicorn for metamonitor_public_endpoint.
Oct 02 06:10:11 alert1002 systemd[1]: metamonitoring_public_endpoint.service: Consumed 13.912s CPU time.
Oct 02 06:10:11 alert1002 systemd[1]: Started metamonitoring_public_endpoint.service - Gunicorn for metamonitor_public_endpoint.
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     sys.stderr.write('--- Logging error ---\n')
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 242, in handle_chld
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.reap_workers()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 530, in reap_workers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.log.warning(
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/glogging.py", line 261, in warning
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.error_log.warning(msg, *args, **kwargs)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1501, in warning
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self._log(WARNING, msg, args, **kwargs)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1634, in _log
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.handle(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1644, in handle
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.callHandlers(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     hdlr.handle(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 978, in handle
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.emit(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1230, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     StreamHandler.emit(self, record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1118, in emit
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.handleError(record)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3.11/logging/__init__.py", line 1031, in handleError
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     sys.stderr.write('--- Logging error ---\n')
Oct 02 06:09:41 alert1002 gunicorn[3038713]: RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Call stack:
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/bin/gunicorn", line 33, in <module>
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     sys.exit(load_entry_point('gunicorn==20.1.0', 'console_scripts', 'gunicorn')())
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 67, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/app/base.py", line 231, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     super().run()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/app/base.py", line 72, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     Arbiter(self).run()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 227, in run
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.halt()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 342, in halt
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.stop()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 393, in stop
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     time.sleep(0.1)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 242, in handle_chld
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.reap_workers()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 513, in reap_workers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     wpid, status = os.waitpid(-1, os.WNOHANG)
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 242, in handle_chld
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.reap_workers()
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 530, in reap_workers
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.log.warning(
Oct 02 06:09:41 alert1002 gunicorn[3038713]:   File "/usr/lib/python3/dist-packages/gunicorn/glogging.py", line 261, in warning
Oct 02 06:09:41 alert1002 gunicorn[3038713]:     self.error_log.warning(msg, *args, **kwargs)
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Message: 'Worker with pid %s was terminated due to signal %s'
Oct 02 06:09:41 alert1002 gunicorn[3038713]: Arguments: (3039077, 15)
Oct 02 06:10:11 alert1002 systemd[1]: metamonitoring_public_endpoint.service: Deactivated successfully.
Oct 02 06:10:11 alert1002 systemd[1]: Stopped metamonitoring_public_endpoint.service - Gunicorn for metamonitor_public_endpoint.
Oct 02 06:10:11 alert1002 systemd[1]: metamonitoring_public_endpoint.service: Consumed 13.912s CPU time.
Oct 02 06:10:11 alert1002 systemd[1]: Started metamonitoring_public_endpoint.service - Gunicorn for metamonitor_public_endpoint.

Some minutes before and some minutes after the service worked normally. Also the pa.ge resolved automatically.

gerritbot added a comment.Oct 2 2025, 1:17 PM

Change #1193109 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: replace Gunicorn with uWSGI

https://gerrit.wikimedia.org/r/1193109

gerritbot added a project: Patch-For-Review.Oct 2 2025, 1:17 PM
gerritbot added a comment.Oct 3 2025, 6:00 AM

Change #1193109 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: replace Gunicorn with uWSGI

https://gerrit.wikimedia.org/r/1193109

Maintenance_bot removed a project: Patch-For-Review.Oct 3 2025, 6:30 AM
gerritbot added a comment.Oct 3 2025, 8:59 AM

Change #1193373 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: cleanup gunicorn related resources

https://gerrit.wikimedia.org/r/1193373

gerritbot added a project: Patch-For-Review.Oct 3 2025, 8:59 AM

Change #1193374 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: rename uwsgi resource

https://gerrit.wikimedia.org/r/1193374

gerritbot added a comment.Oct 3 2025, 8:59 AM

Change #1193375 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: prepare deadmanswitchamhook's Gunicorn replacement with uWSGI

https://gerrit.wikimedia.org/r/1193375

gerritbot added a comment.Oct 3 2025, 9:00 AM

Change #1193376 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: replace deadmanswitchamhook's Gunicorn with uWSGI

https://gerrit.wikimedia.org/r/1193376

gerritbot added a comment.Oct 3 2025, 9:46 AM

Change #1193381 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: add env vars to uwsgi process

https://gerrit.wikimedia.org/r/1193381

gerritbot added a comment.Oct 3 2025, 9:46 AM

Change #1193382 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: cleanup unneeded env files

https://gerrit.wikimedia.org/r/1193382

gerritbot added a comment.Oct 3 2025, 9:52 AM

Change #1193373 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: cleanup gunicorn related resources

https://gerrit.wikimedia.org/r/1193373

gerritbot added a comment.Oct 3 2025, 10:00 AM

Change #1193374 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: rename uwsgi resource

https://gerrit.wikimedia.org/r/1193374

gerritbot added a comment.Oct 3 2025, 10:05 AM

Change #1193375 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: prepare deadmanswitchamhook's Gunicorn replacement with uWSGI

https://gerrit.wikimedia.org/r/1193375

gerritbot added a comment.Oct 3 2025, 10:13 AM

Change #1193376 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: replace deadmanswitchamhook's Gunicorn with uWSGI

https://gerrit.wikimedia.org/r/1193376

gerritbot added a comment.Oct 3 2025, 10:18 AM

Change #1193381 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: add env vars to uwsgi process

https://gerrit.wikimedia.org/r/1193381

gerritbot added a comment.Oct 3 2025, 10:24 AM

Change #1193382 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: cleanup unneeded env files

https://gerrit.wikimedia.org/r/1193382

Maintenance_bot removed a project: Patch-For-Review.Oct 3 2025, 10:32 AM
gerritbot added a comment.Oct 7 2025, 7:53 AM

Change #1194121 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: avoid unnecessary public endpoint restarts

https://gerrit.wikimedia.org/r/1194121

gerritbot added a project: Patch-For-Review.Oct 7 2025, 7:53 AM
gerritbot added a comment.Oct 7 2025, 8:55 AM

Change #1194121 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: avoid unnecessary public endpoint restarts

https://gerrit.wikimedia.org/r/1194121

Maintenance_bot removed a project: Patch-For-Review.Oct 7 2025, 9:31 AM
gerritbot added a comment.Nov 7 2025, 10:38 AM

Change #1202990 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[labs/private@master] metamonitoring/icinga/ext-mon: add dummy basic auth info

https://gerrit.wikimedia.org/r/1202990

gerritbot added a project: Patch-For-Review.Nov 7 2025, 10:38 AM
gerritbot added a comment.Nov 7 2025, 10:40 AM

Change #1202990 merged by Tiziano Fogli:

[labs/private@master] metamonitoring/icinga/ext-mon: add dummy basic auth info

https://gerrit.wikimedia.org/r/1202990

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2025, 11:31 AM
gerritbot added a comment.Nov 10 2025, 10:51 AM

Change #1203405 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[labs/private@master] metamonitoring/icinga/ext-mon: add dummy basic auth info

https://gerrit.wikimedia.org/r/1203405

gerritbot added a project: Patch-For-Review.Nov 10 2025, 10:51 AM

Change #1203405 merged by Tiziano Fogli:

[labs/private@master] metamonitoring/icinga/ext-mon: add dummy basic auth info

https://gerrit.wikimedia.org/r/1203405

Maintenance_bot removed a project: Patch-For-Review.Nov 10 2025, 11:32 AM
gerritbot added a comment.Nov 11 2025, 4:00 PM

Change #1203845 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] metamonitoring: add icinga module

https://gerrit.wikimedia.org/r/1203845

gerritbot added a project: Patch-For-Review.Nov 11 2025, 4:00 PM
gerritbot added a comment.Nov 12 2025, 9:31 AM

Change #1203845 merged by Tiziano Fogli:

[operations/puppet@production] metamonitoring: add icinga module

https://gerrit.wikimedia.org/r/1203845

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2025, 10:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits