WMCS galera cluster checks failing from new alert[12]* hosts
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Aug 18 2020, 1:13 PM

Description

We'll be putting in service new icinga hosts (alert[12]*, addresses below) and currently the WMCS Galera checks are failing. I'm assuming because grants need to be adjusted, e.g.

[1597756054] SERVICE ALERT: cloudcontrol2001-dev;WMCS Galera Cluster #page;WARNING;HARD;3;User does not have access privileges to database: Access denied for user 'galera_monitor'@'208.80.154.88' (using password: YES)
[1597756054] SERVICE ALERT: cloudcontrol2003-dev;WMCS Galera Cluster #page;WARNING;HARD;3;User does not have access privileges to database: Access denied for user 'galera_monitor'@'208.80.154.88' (using password: YES)
[1597756054] SERVICE ALERT: cloudcontrol2004-dev;WMCS Galera Cluster #page;WARNING;HARD;3;User does not have access privileges to database: Access denied for user 'galera_monitor'@'208.80.154.88' (using password: YES)

The new hosts are

alert1001.wikimedia.org has address 208.80.154.88
alert1001.wikimedia.org has IPv6 address 2620:0:861:3:208:80:154:88

alert2001.wikimedia.org has address 208.80.153.84
alert2001.wikimedia.org has IPv6 address 2620:0:860:3:208:80:153:84

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	wmcs galera tests: don't page the whole SRE team for galera issues!	operations/puppet	production	+0 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	dancy	T302086 Set scap minimum python version to 3.7
Resolved	None	T247045 Migrate all of production metal and VMs to Buster or later
Resolved	• lmata	T247966 Migrate role::alerting_host to Buster
Resolved	herron	T260688 WMCS galera cluster checks failing from new alert[12]* hosts

Event Timeline

fgiunchedi created this task.Aug 18 2020, 1:13 PM

Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptAug 18 2020, 1:13 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

on cloudcontrol1003.wikimedia.org:

root@cloudcontrol1003:/etc/nova# mysql -u root
mysql:root@localhost [(none)]> GRANT USAGE ON *.* TO 'galera_monitor'@'208.80.154.88' IDENTIFIED BY '<redacted>';
mysql:root@localhost [(none)]> GRANT USAGE ON *.* TO 'galera_monitor'@'208.80.153.84' IDENTIFIED BY '<redacted>';
mysql:root@localhost [(none)]> GRANT USAGE ON *.* TO 'galera_monitor'@'address 2620:0:861:3:208:80:154:88' IDENTIFIED BY '<redacted>';
mysql:root@localhost [(none)]> GRANT USAGE ON *.* TO 'galera_monitor'@'2620:0:860:3:208:80:153:84' IDENTIFIED BY '<redacted>';

Change 622613 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs galera tests: don't page the whole SRE team for galera issues!

https://gerrit.wikimedia.org/r/622613

Change 622613 merged by Andrew Bogott:
[operations/puppet@production] wmcs galera tests: don't page the whole SRE team for galera issues!

https://gerrit.wikimedia.org/r/622613