Page MenuHomePhabricator
Create Task
Maniphest T109732

Add monitoring for expected load issues on tool labs exec nodes
Closed, DeclinedPublic
Actions

Description

See also T90542

We currently have no monitoring for availability of exec nodes. This means we are not notified if a queue is overwhelmed and e.g. no more webservice processes can start.

See also https://wikitech.wikimedia.org/wiki/Incident_documentation/20150817-ToolLabs-WebgridOutage

Related Objects

Event Timeline

valhallasw created this task.Aug 20 2015, 4:45 PM
valhallasw raised the priority of this task from to Needs Triage.
valhallasw updated the task description. (Show Details)
valhallasw added a project: Toolforge.
valhallasw subscribed.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 20 2015, 4:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
scfc subscribed.Aug 20 2015, 5:03 PM

In T50668, I suggested for that:

  • Count of jobs in error state doesn't exceed 5 % of all jobs running,
  • count of jobs pending doesn't exceed 5 % of all jobs running.
valhallasw triaged this task as High priority.Oct 4 2015, 11:39 AM
valhallasw set Security to None.
valhallasw moved this task from Backlog to Ready to be worked on on the Toolforge board.
valhallasw added a subscriber: yuvipanda.
• chasemp renamed this task from Add monitoring for expected load issues to Add monitoring for expected load issues on tool labs exec nodes.Nov 30 2015, 6:42 PM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:47 PM
Krinkle edited projects, added Sustainability (Incident Followup); removed Cloud-Services.Sep 28 2021, 8:09 PM
dcaro closed this task as Declined.Jan 24 2024, 1:49 PM
dcaro subscribed.

The Grid is going away soon, no need to add more monitoring.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits