Page MenuHomePhabricator
Create Task
Maniphest T292820

Create runbook for VarnishTrafficDrop alert, change dashboard link
Closed, ResolvedPublic
Actions

Description

The VarnishTrafficDrop alert currently has no associated runbook, making it hard for SREs to know what to do about it.

Also, the dashboard currently linked is useful but varnish-caching-last-week-comparison, specific to the affected DC if possible, would be better.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
VarnishTrafficDrop: add runbook and change dashboard linkoperations/alertsmaster+4 -2
Customize query in gerrit

Related Objects

Event Timeline

• ema created this task.Oct 8 2021, 9:02 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 8 2021, 9:02 AM
• ema triaged this task as Medium priority.Oct 8 2021, 11:53 AM
BBlack moved this task from Backlog to Scheduled incidental work on the Traffic board.Oct 8 2021, 7:51 PM
• ema moved this task from Backlog to Doing on the User-ema board.Oct 12 2021, 11:38 AM
• ema added a comment.Oct 12 2021, 12:18 PM

Runbook created: https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop

gerritbot added a comment.Oct 12 2021, 12:24 PM

Change 730193 had a related patch set uploaded (by Ema; author: Ema):

[operations/alerts@master] VarnishTrafficDrop: add runbook and change dashboard link

https://gerrit.wikimedia.org/r/730193

gerritbot added a project: Patch-For-Review.Oct 12 2021, 12:24 PM
gerritbot added a comment.Oct 12 2021, 3:10 PM

Change 730193 merged by Ema:

[operations/alerts@master] VarnishTrafficDrop: add runbook and change dashboard link

https://gerrit.wikimedia.org/r/730193

• ema mentioned this in rOALE3843b0bc4c70: VarnishTrafficDrop: add runbook and change dashboard link.Oct 12 2021, 3:10 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 12 2021, 3:10 PM
• ema closed this task as Resolved.Oct 13 2021, 9:24 AM
• ema claimed this task.

Runbook and updated dashboard link are shown correctly. Closing.

06:00 < jinxer-wm> (VarnishTrafficDrop) resolved: 68% GET drop in text@codfw during the past 30 minutes - 
                   https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop - 
                   https://grafana.wikimedia.org/d/000000541/varnish-caching-last-week-comparison?viewPanel=5&var-cluster=text&var-site=codfw - 
                   https://alerts.wikimedia.org
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits