Change 398533 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] maintain-dbusers: Stop managing account creation for labsdb1001 and 1003

https://gerrit.wikimedia.org/r/398533

Change 398533 merged by Madhuvishy:
[operations/puppet@production] maintain-dbusers: Stop managing account creation for labsdb1001 and 1003

https://gerrit.wikimedia.org/r/398533

I dropped all the status=absent accounts from labsdbaccount.account_host for labsdb1001 and 1003.

What is the state of this now due to T184832?

@jcrespo I'd like to drop all the accounts metadata for labsdb1001 & 3 from labsdbaccounts.account_host on m5-master to close this task.

That would mean running:

delete from labsdbaccounts.account_host where hostname='labsdb1001.eqiad.wmnet';
delete from labsdbaccounts.account_host where hostname='labsdb1003.eqiad.wmnet';

I'm happy to back up the data first before running this. Does this sound good to you?

@madhuvishy data management of m* databases is something I do not handle or you do not have to ask permission for. My suggestion is to connect as the user of the service, and not as root (as to minimize mistakes), and just run those commands. We already perform backups weekly, but apparently this new db was not added. This should be a blocker of your maintenance there.

This is the new list of m5 databases being backed up- if you confirm that is as intended, you can proceed now we have proper[sic] backups.

root@dbstore2001:/srv/backups/m5.20180214102352$ ls *-schema-create.sql.gz
ceilometer-schema-create.sql.gz
designate_pool_manager-schema-create.sql.gz
designate-schema-create.sql.gz
glance-schema-create.sql.gz
keystone-schema-create.sql.gz
labsdbaccounts-schema-create.sql.gz
labspuppet-schema-create.sql.gz
neutron-schema-create.sql.gz
nodepooldb-schema-create.sql.gz
nova-schema-create.sql.gz
striker-schema-create.sql.gz

In T183029#3971300, @jcrespo wrote:

This is the new list of m5 databases being backed up- if you confirm that is as intended, you can proceed now we have proper[sic] backups.

root@dbstore2001:/srv/backups/m5.20180214102352$ ls *-schema-create.sql.gz
ceilometer-schema-create.sql.gz
designate_pool_manager-schema-create.sql.gz
designate-schema-create.sql.gz
glance-schema-create.sql.gz
keystone-schema-create.sql.gz
labsdbaccounts-schema-create.sql.gz
labspuppet-schema-create.sql.gz
neutron-schema-create.sql.gz
nodepooldb-schema-create.sql.gz
nova-schema-create.sql.gz
striker-schema-create.sql.gz

@jcrespo That looks good, thank you!

I've dropped the metadata for labsdb1001 and 1003 from labsdbaccounts.account_host. It now looks like

madhuvishy@labstore1004:~$ mysql -h m5-master.eqiad.wmnet -u labsdbaccounts -p -e "select hostname, count(*) from labsdbaccounts.account_host group by hostname;"
+------------------------+----------+
| hostname               | count(*) |
+------------------------+----------+
| labsdb1005.eqiad.wmnet |     3115 |
| labsdb1009.eqiad.wmnet |     3114 |
| labsdb1010.eqiad.wmnet |     3114 |
| labsdb1011.eqiad.wmnet |     3114 |
+------------------------+----------+