Page MenuHomePhabricator
Create Task
Maniphest T319350

nbserve high load
Closed, ResolvedPublic
Actions

Description

Something is hammering nbserve

nbserve-6dcb78656d-h9qk5                   7684m        4740Mi

Which might explain jupyterhub the alerts that have been coming through:

(PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
4:47 PM (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown

Related Objects

Event Timeline

rook created this task.Oct 4 2022, 8:57 PM
rook added a comment.Oct 4 2022, 9:00 PM

Pod restart seemed to get it to calm down. Will check in the morning.

rook added a comment.Oct 5 2022, 4:36 AM

nbserve still seems fine. Though the jupyterhub alerts are still coming through.

rook added a comment.Oct 5 2022, 4:43 AM

IRC logs showing flapping.

[2022-10-04 21:04:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 21:09:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 21:29:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 21:34:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 21:35:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 21:40:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 22:13:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 22:33:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 22:37:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 22:47:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 22:55:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:00:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:15:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:20:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:25:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:30:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:32:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:37:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:46:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-04 23:56:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 00:01:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 00:06:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 00:21:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 00:26:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 00:43:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 00:48:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 00:59:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 01:19:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 01:35:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 01:40:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 02:21:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 02:26:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 02:32:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 02:37:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 04:34:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 04:39:50] <wmcs-alerts> (PawsJupyterHubDown) resolved: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[2022-10-05 04:54:50] <wmcs-alerts> (PawsJupyterHubDown) firing: PAWS JupyterHub is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
rook added a comment.Oct 5 2022, 4:45 AM

Flapping investigated in T319366

rook closed this task as Resolved.Oct 5 2022, 4:45 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits