Change Details

In {T250206} we worked on a proof of concept project to design a replacement for monitoring a subset of projects using Shinken with Prometheus. Later we realized that {T210993} means that we also need to migrate to Prometheus for basic dashboards like {T264920}. We now need to redesign and build out the POC to scale to collecting at least basic instance health information for all instances in all projects for some reasonable amount of time (3+ months for sure, 1+ year ideally). Features needed beyond POC: [] Storage for metrics from 700+ instances [x] Redundancy [] Configure the alert rules to monitor disk capacity [x] Project local puppetmaster for secrets storage (e.g. volunteer's email addresses) [] ~Refactor puppet module to merge public and private hiera config~ [x] irc relay for irc alerting per project [x] karma alert management dashboard with ability to silence alerts (T250206#6443507, {T285055})

In {T250206} we worked on a proof of concept project to design a replacement for monitoring a subset of projects using Shinken with Prometheus. Later we realized that {T210993} means that we also need to migrate to Prometheus for basic dashboards like {T264920}. We now need to redesign and build out the POC to scale to collecting at least basic instance health information for all instances in all projects for some reasonable amount of time (3+ months for sure, 1+ year ideally). Features needed beyond POC: [x] Storage for metrics from 700+ instances [x] Redundancy [] Configure the alert rules to monitor disk capacity [x] Project local puppetmaster for secrets storage (e.g. volunteer's email addresses) [] ~~Refactor puppet module to merge public and private hiera config~~ [x] irc relay for irc alerting per project [x] karma alert management dashboard with ability to silence alerts (T250206#6443507, {T285055})

In {T250206} we worked on a proof of concept project to design a replacement for monitoring a subset of projects using Shinken with Prometheus. Later we realized that {T210993} means that we also need to migrate to Prometheus for basic dashboards like {T264920}. We now need to redesign and build out the POC to scale to collecting at least basic instance health information for all instances in all projects for some reasonable amount of time (3+ months for sure, 1+ year ideally). Features needed beyond POC: [x] Storage for metrics from 700+ instances [x] Redundancy [] Configure the alert rules to monitor disk capacity [x] Project local puppetmaster for secrets storage (e.g. volunteer's email addresses) [] ~~Refactor puppet module to merge public and private hiera config~~ [x] irc relay for irc alerting per project [x] karma alert management dashboard with ability to silence alerts (T250206#6443507, {T285055})