Page MenuHomePhabricator

Setup Fundraising team in VO/splunk oncall
Closed, ResolvedPublic

Description

This task is in the context of fully migrating off the old paging system (i.e. SMS sent by icinga via a 3rd party provider) and onto VictorOps / Splunk Oncall.

For Fundraising tech specifically we'll need to at do/coordinate these steps:

  • Create a new team in VO (preferred name? Fundraising ?) and add @Jgreen + @Dwisehaupt (already onboarded in VO)
  • Create a rotation and escalation for the team, and assign a new routing key (for the icinga integration)
  • Define a new VO contact in icinga victorops-fundraising
  • Add victorops-fundraising contact to fr-tech-ops contact in parallel (for validation)
  • Remove individual sms contacts from fr-tech-ops
  • Make sure team onboarding documentation is updated to reflect VO onboarding and stop changing icinga contacts.cfg file

Related Objects

StatusSubtypeAssignedTask
Resolvedherron

Event Timeline

Change 662996 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] icinga: send fr-tech-ops alerts to victorops-fundraising

https://gerrit.wikimedia.org/r/662996

Change 662996 merged by Jgreen:
[operations/puppet@production] icinga: send fr-tech-ops alerts to victorops-fundraising

https://gerrit.wikimedia.org/r/662996

Change 670927 had a related patch set uploaded (by Jgreen; owner: Jgreen):
[operations/puppet@production] remove parallel notification of jgreen/dwisehaupt for fr-tech-ops icinga alerts

https://gerrit.wikimedia.org/r/670927

Change 670927 merged by Jgreen:
[operations/puppet@production] remove parallel notification of jgreen/dwisehaupt for fr-tech-ops icinga alerts

https://gerrit.wikimedia.org/r/670927

I've also removed notify-by-sms-gateway notification commands configuration for fr contacts in private.git, I believe we're all set on the technical front! Last bit is to make sure onboarding mentions VO/Splunk Oncall as opposed to Icinga contacts. Thanks all!

@Jgreen @Dwisehaupt I couldn't find fr-tech oncall onboarding documentation that mentions Icinga SMS vs VO/Splunk Oncall, is there anything else left to do ?

@fgiunchedi We don't have any separate documentation relevant to on-call, and would just follow the regular SRE docs/procedures.

@fgiunchedi We don't have any separate documentation relevant to on-call, and would just follow the regular SRE docs/procedures.

Ack, thank you! Resolving the task since I believe we're all done