Page MenuHomePhabricator

Ensure Data Platform SREs have a contact group in puppet/alerting
Closed, ResolvedPublic

Assigned To
Authored By
bking
Jul 24 2023, 8:19 PM
Referenced Files
F41815951: image.png
Feb 8 2024, 3:09 PM
F41667856: image.png
Jan 12 2024, 5:38 PM
F41666018: image.png
Jan 11 2024, 5:12 PM

Description

Before we can fully assume responsibility for services, the Data Platform team needs to add and verify our contacts and alerting.

Creating this ticket to

  • Add relevant puppet code for this new team
  • Confirm the above is working as expected

Note: T296552 is related to cleaning up our alerting

Event Timeline

As discussed in DPE SRE meeting:

  • let's try to be consistent in naming, using "Data Platform" for all services where DPE SRE is responsible
Gehel triaged this task as High priority.Nov 22 2023, 9:44 AM
Gehel moved this task from Misc to Ready for Work on the Data-Platform-SRE board.

Change 989900 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] WIP: Add new data platform team to alertmanager

https://gerrit.wikimedia.org/r/989900

I have renamed the Data Engineering team (https://portal.victorops.com/dash/wikimedia#/team-schedules) in VictorOps to Data Platform SRE.

image.png (1,046×402 px, 39 KB)

I have added @bking and @RKemper to it, plus I've sent an invitation to join Victorops to @brouberol.

(edit) Oops, sorry. This comment should probably have ben on the sub-ticket: T344202

Thanks @BTullis! I've created the new data-platform-alerts@lists.wikimedia.org mailing list . I created it using the CLI as described here . I don't appear to be an admin in the mailing list web UI, so I might need some help setting the description and adding myself and others as list administrators.

Change 998796 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a new Icinga contact group for team-data-platform

https://gerrit.wikimedia.org/r/998796

@bking - I hope you don't mind, but I'm going to claim this ticket for a little bit. Recent changes to the alertmanager config have begun making life difficult for the Ops Week engineers by overwhelming email as well as IRC. I'm going to try to alleviate this pressure a little, if I can.

Change 998826 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update default role contactgroups from analytics to team-data-platform

https://gerrit.wikimedia.org/r/998826

Change 998838 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add icinga commands to notify the Data Platform SRE team

https://gerrit.wikimedia.org/r/998838

Change 998842 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update contactgroups for analytics => team-data-platform icinga checks

https://gerrit.wikimedia.org/r/998842

Change 998838 merged by Btullis:

[operations/puppet@production] Add icinga configuration to notify the Data Platform SRE team

https://gerrit.wikimedia.org/r/998838

Change 998796 merged by Btullis:

[operations/puppet@production] Add a new Icinga contact group for team-data-platform

https://gerrit.wikimedia.org/r/998796

Change 998826 merged by Btullis:

[operations/puppet@production] Update default role contactgroups from analytics to team-data-platform

https://gerrit.wikimedia.org/r/998826

I have modified the group settings for data-platform-alerts@wikimedia.org so that anyone on the web can post. I believe that this is required in order to accept email from Icinga/Alertmanager.

image.png (798×861 px, 108 KB)

Change 998842 merged by Btullis:

[operations/puppet@production] Update contactgroups for analytics => team-data-platform icinga checks

https://gerrit.wikimedia.org/r/998842

Change 989900 merged by Btullis:

[operations/puppet@production] Onboard the data-platform-sre team to Alertmanager

https://gerrit.wikimedia.org/r/989900

Change 999561 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Change all role contacts for Data Engineering -> Data Platform

https://gerrit.wikimedia.org/r/999561

Change 999561 merged by Btullis:

[operations/puppet@production] Change all role contacts for Data Engineering -> Data Platform

https://gerrit.wikimedia.org/r/999561

Change 999858 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alertmanager: add data-engineering-mail missing receiver

https://gerrit.wikimedia.org/r/999858

Change 999858 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: add data-engineering-mail missing receiver

https://gerrit.wikimedia.org/r/999858