Page MenuHomePhabricator
Create Task
Maniphest T137869

Review alerting scheme for Maps
Closed, ResolvedPublic
Actions

Description

Before calling maps production ready, we need to review alerting scheme and ensure we page the right people when something goes wrong and we don't wake up anyone if things are not that bad.

Summary of current alerts:

Critical services
Those services are directly used to serve content to users

ServiceCheck descriptionContact groups
cassandraTCP check on Cassandra portadmins,team-services
cassandraService checkadmins
kartotherianService checker (Swagger based)admins
maps5xx rateadmins
maps-lbHTTP checks on HTTP, HTTPS, IPv4 and IPv6 for each DCadmins,sms,admins
varnishstandard varnish checksadmins
kartotherian LVSLVS checkadmins,sms,admins

Non critical services
Those services are use for tile generation, they have no direct user impact

ServiceCheck descriptionContact groups
tileratorHTTP checkadmins
tileratoruiHTTP checkadmins
postgresno check yet

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Team-interactive receives maps alertsoperations/puppetproduction+67 -15
Add interactive-team to default Icinga notification group for maps serversoperations/puppetproduction+3 -0
Add the ability to configure contact group for check of services.operations/puppetproduction+23 -7
Interactive team would like to be notified of issues with Maps.operations/puppetproduction+1 -1
Adding a "interactive-team" icinga group for alerting.operations/puppetproduction+24 -7
Customize query in gerrit

Related Objects
Search...

Event Timeline

Gehel created this task.Jun 15 2016, 7:39 AM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 15 2016, 7:39 AM
Gehel updated the task description. (Show Details)Jun 15 2016, 9:59 AM
Gehel added a subscriber: MaxSem.Jun 15 2016, 10:02 AM

We should probably create a maps team with @Yurik, @MaxSem for kartotherian and tilerator alerts.

Joe subscribed.Jun 15 2016, 1:24 PM

We should add a service check for karthoterian using service_checker on the lvs IP, pretty much as we do for other services, see:

https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/lvs/manifests/monitor_services.pp

and probably we should make it page too.

As far as paging goes, I'd stick with having public-facing services paging us (so just karthoterian on the LVS).

gerritbot subscribed.Jun 15 2016, 2:22 PM

Change 294498 had a related patch set uploaded (by Gehel):
Adding a "interactive-team" icinga group for alerting.

https://gerrit.wikimedia.org/r/294498

gerritbot added a project: Patch-For-Review.Jun 15 2016, 2:22 PM
Gehel added a comment.Jun 15 2016, 2:44 PM

LVS endpoint are checked. The check_http_lvs_on_port check does page.

gerritbot added a comment.Jun 15 2016, 3:08 PM

Change 294503 had a related patch set uploaded (by Gehel):
Add interactive-team to default Icinga notification group for maps servers

https://gerrit.wikimedia.org/r/294503

gerritbot added a comment.Jun 15 2016, 3:37 PM

Change 294507 had a related patch set uploaded (by Gehel):
Add the ability to configure contact group for check of services.

https://gerrit.wikimedia.org/r/294507

Gehel moved this task from Backlog to In progress on the Maps-Sprint board.Jun 15 2016, 3:38 PM
Yurik added a comment.Jun 15 2016, 6:11 PM

@Gehel, what does admins,sms,admins mean? I think the first 4 items are relevant to @MaxSem and myself - Cassandra not working, Kartotherian is down, or too many 5xx

Gehel added a comment.Jun 15 2016, 8:18 PM

@Yurik admins,sms,admins are the groups to which the alerts are sent (I have no idea why admins is there twice). admins is the ops group, sms is the pagers.

I'll check how to add you to those alerts.

gerritbot added a comment.Jun 15 2016, 9:09 PM

Change 294498 merged by Gehel:
Adding a "interactive-team" icinga group for alerting.

https://gerrit.wikimedia.org/r/294498

Gehel mentioned this in rOPUP760787c5c7b8: Adding a "interactive-team" icinga group for alerting..Jun 15 2016, 9:11 PM
gerritbot added a comment.Jun 16 2016, 9:13 AM

Change 294676 had a related patch set uploaded (by Gehel):
Interactive team would like to be notified of issues with Maps.

https://gerrit.wikimedia.org/r/294676

gerritbot added a comment.Jun 16 2016, 9:28 AM

Change 294676 merged by Gehel:
Interactive team would like to be notified of issues with Maps.

https://gerrit.wikimedia.org/r/294676

Gehel mentioned this in rOPUP69e9480ccd8e: Interactive team would like to be notified of issues with Maps..Jun 16 2016, 9:31 AM
gerritbot added a comment.Jun 16 2016, 1:10 PM

Change 294507 merged by Gehel:
Add the ability to configure contact group for check of services.

https://gerrit.wikimedia.org/r/294507

Gehel mentioned this in rOPUP65a8dcc4f4b5: Add the ability to configure contact group for check of services..Jun 16 2016, 1:14 PM
Gehel added a comment.Jun 16 2016, 2:49 PM
In T137869#2382475, @Joe wrote:

We should add a service check for karthoterian using service_checker on the lvs IP, pretty much as we do for other services, see:

https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/lvs/manifests/monitor_services.pp

and probably we should make it page too.

As far as paging goes, I'd stick with having public-facing services paging us (so just karthoterian on the LVS).

That suggestion looks good to me. I'll add as well team-interactive to those alerts.

Gehel mentioned this in T137851: "Is maps service alive?" check.Jun 16 2016, 3:05 PM
gerritbot added a comment.Jun 16 2016, 3:20 PM

Change 294723 had a related patch set uploaded (by Gehel):
Team-interactive receives maps alerts

https://gerrit.wikimedia.org/r/294723

Gehel added a comment.Jun 16 2016, 3:35 PM

In term of production support, we seem to be good to go once https://gerrit.wikimedia.org/r/#/c/294723/ is merged. LVS will be paging.

We can do some more improvement in term of monitoring / alerting (monitor better non critical services like Postgres), but that is outside of the scope of this ticket. See T135647 instead.

gerritbot added a comment.Jun 16 2016, 4:27 PM

Change 294503 abandoned by Gehel:
Add interactive-team to default Icinga notification group for maps servers

Reason:
replaced by https://gerrit.wikimedia.org/r/#/c/294723/ which give more finer grain alerting to team-interactive

https://gerrit.wikimedia.org/r/294503

gerritbot added a comment.Jun 16 2016, 8:39 PM

Change 294723 merged by Gehel:
Team-interactive receives maps alerts

https://gerrit.wikimedia.org/r/294723

Gehel mentioned this in rOPUP94f4d523b5eb: Team-interactive receives maps alerts.Jun 16 2016, 8:41 PM
Gehel closed this task as Resolved.Jun 16 2016, 8:43 PM
Gehel mentioned this in rOPUP9f9ad82f3f6a: Team-interactive receives maps alerts.Jun 17 2016, 6:05 PM
Gehel mentioned this in rOPUPaa7a082564ee: Team-interactive receives maps alerts.
Gehel mentioned this in rOPUPc1cfc5e92818: Team-interactive receives maps alerts.
Gehel mentioned this in rOPUPa9ab56c35037: Team-interactive receives maps alerts.
Gehel mentioned this in rOPUP6f4a877f095d: Team-interactive receives maps alerts.
Gehel mentioned this in rOPUPf3db26f0380f: Team-interactive receives maps alerts.
Gehel mentioned this in rOPUP24aa3327f8eb: Team-interactive receives maps alerts.
Gehel mentioned this in rOPUP245fc03026d7: Interactive team would like to be notified of issues with Maps..
Gehel mentioned this in rOPUP0726fc3c13c5: Add the ability to configure contact group for check of services..
Gehel mentioned this in rOPUP2f5782ef74bb: Adding a "interactive-team" icinga group for alerting..
Gehel mentioned this in rOPUPc0c928e53597: Add interactive-team to default Icinga notification group for maps servers.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits