Page MenuHomePhabricator

WMCS galera cluster checks failing from new alert[12]* hosts
Closed, ResolvedPublic

Description

We'll be putting in service new icinga hosts (alert[12]*, addresses below) and currently the WMCS Galera checks are failing. I'm assuming because grants need to be adjusted, e.g.

[1597756054] SERVICE ALERT: cloudcontrol2001-dev;WMCS Galera Cluster #page;WARNING;HARD;3;User does not have access privileges to database: Access denied for user 'galera_monitor'@'208.80.154.88' (using password: YES)
[1597756054] SERVICE ALERT: cloudcontrol2003-dev;WMCS Galera Cluster #page;WARNING;HARD;3;User does not have access privileges to database: Access denied for user 'galera_monitor'@'208.80.154.88' (using password: YES)
[1597756054] SERVICE ALERT: cloudcontrol2004-dev;WMCS Galera Cluster #page;WARNING;HARD;3;User does not have access privileges to database: Access denied for user 'galera_monitor'@'208.80.154.88' (using password: YES)

The new hosts are

alert1001.wikimedia.org has address 208.80.154.88
alert1001.wikimedia.org has IPv6 address 2620:0:861:3:208:80:154:88

alert2001.wikimedia.org has address 208.80.153.84
alert2001.wikimedia.org has IPv6 address 2620:0:860:3:208:80:153:84

Event Timeline

Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptAug 18 2020, 1:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Andrew added a subscriber: Andrew.Wed, Aug 26, 4:47 PM

on cloudcontrol1003.wikimedia.org:

root@cloudcontrol1003:/etc/nova# mysql -u root
mysql:root@localhost [(none)]> GRANT USAGE ON *.* TO 'galera_monitor'@'208.80.154.88' IDENTIFIED BY '<redacted>';
mysql:root@localhost [(none)]> GRANT USAGE ON *.* TO 'galera_monitor'@'208.80.153.84' IDENTIFIED BY '<redacted>';
mysql:root@localhost [(none)]> GRANT USAGE ON *.* TO 'galera_monitor'@'address 2620:0:861:3:208:80:154:88' IDENTIFIED BY '<redacted>';
mysql:root@localhost [(none)]> GRANT USAGE ON *.* TO 'galera_monitor'@'2620:0:860:3:208:80:153:84' IDENTIFIED BY '<redacted>';

Change 622613 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs galera tests: don't page the whole SRE team for galera issues!

https://gerrit.wikimedia.org/r/622613

Change 622613 merged by Andrew Bogott:
[operations/puppet@production] wmcs galera tests: don't page the whole SRE team for galera issues!

https://gerrit.wikimedia.org/r/622613

herron closed this task as Resolved.Wed, Aug 26, 5:03 PM
herron claimed this task.

Thanks! These Galera checks are now green in the new icinga instance