Page MenuHomePhabricator
Create Task
Maniphest T134551

Create functional cluster checks for all services (and have them page!)
Closed, ResolvedPublic
Actions

Description

Service_checker does a good job of verifying the functionality of a service on a single machine. During the restbase/parsoid/ve outage we received no pages, as we have not functional test on the cluster as a whole, and we don't page on single-machine outages (rightly so).

Ideas on how to implement functional testing:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20160505-ChangeProp_RESTBase_Parsoid

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
monitoring: use service_checker for mobileapps LVSoperations/puppetproduction+1 -1
lvs::monitor: monitor all services via service_checkeroperations/puppetproduction+105 -18
mobileapps: add experimental cluster checkoperations/puppetproduction+20 -0
nagios_common: Add command for using service_checkeroperations/puppetproduction+17 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Joe created this task.May 6 2016, 5:05 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 6 2016, 5:05 AM
• GWicke mentioned this in T134537: ChangeProp / RESTBase / Parsoid outage 2016-05-05.May 6 2016, 4:51 PM
Joe claimed this task.May 10 2016, 9:01 AM
gerritbot subscribed.May 10 2016, 9:09 AM

Change 287907 had a related patch set uploaded (by Giuseppe Lavagetto):
nagios_common: Add command for using service_checker

https://gerrit.wikimedia.org/r/287907

gerritbot added a project: Patch-For-Review.May 10 2016, 9:09 AM

Change 287908 had a related patch set uploaded (by Giuseppe Lavagetto):
monitoring: use service_checker for mobileapps LVS

https://gerrit.wikimedia.org/r/287908

gerritbot added a comment.May 11 2016, 2:40 PM

Change 287907 merged by Giuseppe Lavagetto:
nagios_common: Add command for using service_checker

https://gerrit.wikimedia.org/r/287907

Joe mentioned this in rOPUPa1e18f824581: nagios_common: Add command for using service_checker.May 11 2016, 2:44 PM
gerritbot added a comment.May 13 2016, 10:31 AM

Change 288589 had a related patch set uploaded (by Giuseppe Lavagetto):
mobileapps: add experimental cluster check

https://gerrit.wikimedia.org/r/288589

gerritbot added a comment.May 13 2016, 10:41 AM

Change 288589 merged by Giuseppe Lavagetto:
mobileapps: add experimental cluster check

https://gerrit.wikimedia.org/r/288589

Joe mentioned this in rOPUP9587be0dde18: mobileapps: add experimental cluster check.May 13 2016, 10:45 AM
Joe added a comment.May 13 2016, 11:11 AM

Mobileapps is now monitored via service_checker on the LVS IPs, and will send emails to the service team and alert on IRC in #-operations

gerritbot added a comment.May 17 2016, 8:23 AM

Change 289151 had a related patch set uploaded (by Giuseppe Lavagetto):
lvs::monitor: monitor all services via service_checker

https://gerrit.wikimedia.org/r/289151

gerritbot added a comment.May 17 2016, 8:40 AM

Change 289151 merged by Giuseppe Lavagetto:
lvs::monitor: monitor all services via service_checker

https://gerrit.wikimedia.org/r/289151

Joe mentioned this in rOPUPf958476f5f12: lvs::monitor: monitor all services via service_checker.May 17 2016, 8:44 AM
Joe added a comment.May 17 2016, 10:22 AM

All active services are now monitored as mobileapps; we can see in a week how many mails did the services team receive and decide if it's ok to page for these.

• mobrovac added a comment.May 26 2016, 4:34 PM

I think we can consider this resolved now?

• GWicke added a comment.May 27 2016, 4:49 PM

I have not gotten any service-global alerts so far, and would expect them to be very rare in any case. I do get occasional alerts for individual service nodes, typically connected to ongoing work.

Given the low probability & severity of a service-wide outage, I think we should make those alerts paging. Per-node alerts are more frequent and more of an early warning sign, and do not warrant paging in my opinion.

bd808 unsubscribed.May 27 2016, 4:59 PM
• GWicke added a comment.Jun 2 2016, 9:45 PM

We have recently seen some service-wide alerts after deployment issues, and they were all accurate. @Joe, I think we can go ahead & make those paging.

• GWicke mentioned this in T94821: Set up monitoring automation for services.Jun 7 2016, 5:52 AM
Joe mentioned this in rOPUPfc676962f674: lvs::monitor: monitor all services via service_checker.Jun 17 2016, 6:09 PM
Joe mentioned this in rOPUPaf4cae25ba81: mobileapps: add experimental cluster check.
Joe mentioned this in rOPUPe467727b47b3: mobileapps: add experimental cluster check.
Joe mentioned this in rOPUP3b7e2bfe1cda: nagios_common: Add command for using service_checker.
Joe mentioned this in rOPUP91c4dd3ae5aa: monitoring: use service_checker for mobileapps LVS.
Joe mentioned this in rOPUP62cb3c5d10ea: nagios_common: Add command for using service_checker.
Joe mentioned this in rOPUPfc435758f757: monitoring: use service_checker for mobileapps LVS.
greg added a project: Wikimedia-Incident.Jul 27 2016, 10:54 PM
greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.
greg added a comment.Sep 29 2016, 7:41 PM

This follow-up task from an incident report has not been updated recently. If it is no longer valid, please add a comment explaining why. If it is still valid, please prioritize it appropriately relative to your other work. If you have any questions, feel free to ask me (Greg Grossmeier).

• bearND moved this task from Incoming to Tracking on the Mobile-Content-Service board.Oct 3 2016, 6:11 PM
Joe removed Joe as the assignee of this task.Oct 5 2016, 7:54 AM
Joe added a project: User-Joe.
Joe moved this task from Backlog to Blocking others on the User-Joe board.Oct 5 2016, 8:53 AM
• NHarateh_WMF added a project: Product-Infrastructure-Team-Backlog-Deprecated.Apr 25 2017, 12:24 PM
• NHarateh_WMF moved this task from Needs triage to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.Apr 25 2017, 12:25 PM
• NHarateh_WMF moved this task from Tracking to Backlog on the Mobile-Content-Service board.Apr 25 2017, 4:30 PM
• NHarateh_WMF moved this task from Backlog to Incoming on the Mobile-Content-Service board.Apr 25 2017, 4:37 PM
• GWicke edited projects, added Services (attic); removed Services.Jul 12 2017, 5:54 PM

@Joe, has this been resolved by the external checks (including Varnish) that you set up recently?

Jdlrobson removed a project: Patch-For-Review.Aug 15 2017, 6:32 PM
gerritbot added a comment.Dec 4 2017, 8:49 AM

Change 287908 abandoned by Giuseppe Lavagetto:
monitoring: use service_checker for mobileapps LVS

https://gerrit.wikimedia.org/r/287908

akosiaris mentioned this in T189524: Add Reading Infrastructure engineers to contacts for mobileapps.Apr 13 2018, 10:22 AM
• mobrovac edited projects, added Services (watching), SCB; removed Services (attic).Apr 13 2018, 10:47 AM
• mobrovac added a project: Platform Team Legacy (Watching / External).Dec 20 2018, 12:02 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:39 PM
Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
Krinkle updated the task description. (Show Details)Sep 28 2021, 9:04 PM
akosiaris closed this task as Resolved.Mar 14 2023, 5:31 PM
akosiaris claimed this task.
akosiaris subscribed.

No update since 5+ years ago, I am gonna tentatively resolve this, feel free to reopen.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits