Notes:
Apr 24, 2024 | Cloud VPS monitoring systems
Attendees: Arturo Borrero David Caro Francesco Negri Taavi Väänänen

Notes

• https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#metricsinfra:_Monitoring_services_for_Cloud_VPS
  • https://wikitech.wikimedia.org/wiki/Nova_Resource:Metricsinfra/Documentation (a bit more outdated but more in detail and aspirational planning)
• Goal: Self-service monitoring for the CloudVPS offering
  • Alerts are stored in a database so that in the future we can have an API where project admins can add and manage their own alerts
  • Currently not self-service, but we’re already generating the configs (prometheus/alertmanager) from a database
    • Prometheus manager -> http api to query the DB
    • Prometheus configurator -> pulls from the manager and generates the prometheus config and deploys it
• Managed with opentofu from the repo https://gitlab.wikimedia.org/repos/cloud/metricsinfra/tofu-provisioning (details on the next meeting about opentofu)
• Then puppet classes to configure/drift
• Related to https://phabricator.wikimedia.org/T362012 but probably toolforge is special enough that we might want to not merge it directly with cloud vps.
• What if Cloud VPS goes down entirely? Right now we only have an alert if the alertmanager is down. Task to improve the meta-monitoring: https://phabricator.wikimedia.org/T288053
• There is a cronjob in the configurator server that updates the database, to make sure the table for projects matches the data in keystone, then it syncs the list of images with a specific tag to be used for filtering
• We don’t have backups of the Trove database with the alert config.
• Should we create a temporary system to updated the database without using SQL, or should we wait until we build the API that is also available for users?
• Metricsinfra related parent task https://phabricator.wikimedia.org/T336774

Action items
*