While debugging account access to the wiki replicas, I found that the mysql accounts on labsdb1011 are not in sync with the expected metadata state. This could be related to the recent reclone T235016.
Is there a process we can use to resync the user accounts on labsdb1011? I've found a few accounts missing but haven't done a full audit yet.
Here's some information on an example with my account:
Metadata user status
$ mysql -h m5-master.eqiad.wmnet -u labsdbaccounts -p -e "USE labsdbaccounts; SELECT * FROM account WHERE mysql_username='u21436'\G" Enter password: *************************** 1. row *************************** id: 19201 mysql_username: u21436 type: user username: jeh
Metadata account status
$ mysql -h m5-master.eqiad.wmnet -u labsdbaccounts -p -e "USE labsdbaccounts; SELECT * FROM labsdbaccounts.account_host WHERE account_id=19201\G" Enter password: *************************** 1. row *************************** id: 134314 account_id: 19201 hostname: 172.16.7.153 status: present *************************** 2. row *************************** id: 134313 account_id: 19201 hostname: labsdb1009.eqiad.wmnet status: present *************************** 3. row *************************** id: 134316 account_id: 19201 hostname: labsdb1010.eqiad.wmnet status: present *************************** 4. row *************************** id: 134315 account_id: 19201 hostname: labsdb1011.eqiad.wmnet status: present
Labsdb account status
$ mysql -h 172.16.7.153 -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user where User = "u21436"\G' *************************** 1. row *************************** COUNT(User): 1 $ mysql -h labsdb1009.eqiad.wmnet -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user where User = "u21436"\G' *************************** 1. row *************************** COUNT(User): 1 $ mysql -h labsdb1010.eqiad.wmnet -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user where User = "u21436"\G' *************************** 1. row *************************** COUNT(User): 1 $ mysql -h labsdb1011.eqiad.wmnet -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user where User = "u21436"\G' *************************** 1. row *************************** COUNT(User): 0
Labsdb user table count
$ mysql -h labsdb1009.eqiad.wmnet -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user\G' *************************** 1. row *************************** COUNT(User): 3784 $ mysql -h labsdb1010.eqiad.wmnet -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user\G' *************************** 1. row *************************** COUNT(User): 3778 $ mysql -h labsdb1011.eqiad.wmnet -u labsdbadmin -p -e 'SELECT COUNT(User) from mysql.user\G' *************************** 1. row *************************** COUNT(User): 3543