Maniphest T196158

Review Burrow alarms in order to avoid false positives when restarting it
Closed, ResolvedPublic1 Estimated Story Points
Actions

Assigned To

Authored By

	elukey
	Jun 1 2018, 12:35 PM

Description

When we restart Burrow instances there is a brief amount of time in which the metrics go awol causing alarms to fire, especially the Mirror Maker ones. Ideally Burrow should be able to restart without creating any impact to metrics.

Details

	Subject	Repo	Branch	Lines +/-
	profile::kafka::mirror::alerts: tune retries and its interval	operations/puppet	production	+2 -0

Customize query in gerrit

Event Timeline

elukey created this task.Jun 1 2018, 12:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 1 2018, 12:35 PM

elukey updated the task description. (Show Details)Jun 1 2018, 1:11 PM

• fdans triaged this task as Medium priority.Jun 4 2018, 3:53 PM

• fdans moved this task from Incoming to Operational Excellence on the Analytics board.

Change 438243 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kafka::mirror::alerts: tune retries and its interval

https://gerrit.wikimedia.org/r/438243

gerritbot added a project: Patch-For-Review.Jun 8 2018, 1:19 PM

Change 438243 merged by Elukey:
[operations/puppet@production] profile::kafka::mirror::alerts: tune retries and its interval

https://gerrit.wikimedia.org/r/438243

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Jun 8 2018, 1:32 PM

Tested one burrow restart manually, alarms showed up in icinga but cleared up before firing an alarm. This is a good compromise for the moment, we'll revise the alarm when a more powerful and flexible prometheus check metrics script will be available.

elukey moved this task from In Progress to Done on the Analytics-Kanban board.Jun 11 2018, 7:22 AM

• Vvjjkkii renamed this task from Review Burrow alarms in order to avoid false positives when restarting it to hubaaaaaaa.Jul 1 2018, 1:06 AM

• Vvjjkkii removed elukey as the assignee of this task.

• Vvjjkkii raised the priority of this task from Medium to High.

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed subscribers: gerritbot, Aklapper.

• Community_Tech_bot renamed this task from hubaaaaaaa to Review Burrow alarms in order to avoid false positives when restarting it.Jul 1 2018, 6:22 AM

• Community_Tech_bot assigned this task to elukey.

• Community_Tech_bot updated the task description. (Show Details)

• Community_Tech_bot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.

• Community_Tech_bot added subscribers: gerritbot, Aklapper.

CommunityTechBot lowered the priority of this task from High to Medium.Jul 3 2018, 3:25 AM

• Nuria closed this task as Resolved.Aug 22 2018, 10:42 PM

• Nuria set the point value for this task to 1.

Review Burrow alarms in order to avoid false positives when restarting itClosed, ResolvedPublic1 Estimated Story PointsActions

Description

Details

Event Timeline

Review Burrow alarms in order to avoid false positives when restarting it
Closed, ResolvedPublic1 Estimated Story Points
Actions