Page MenuHomePhabricator

Stop managing account creation for labsdb1001 and 1003 through the maintain-dbusers script
Closed, ResolvedPublic

Description

https://github.com/wikimedia/puppet/blob/production/modules/role/files/labs/db/maintain-dbusers.py now manages creating accounts for tools and Toolforge users. To stop managing the account creation for labsdb1001|3, we'll need to

  • Clean up the metadata stores in m5-master.eqiad.wmnet (labsdbaccount.account_host), and remove entries with hostname = labsdb1001 or 1003, and status=absent - this will stop the script from attempting to create accounts in the old replica hosts for any entries with status absent

These two steps will just prevent maintain-dbusers from attempting to create new accounts in the now-readonly replicas 1001 and 1003.

After the decommission, we should clean up all the entries with hostname = labsdb1001 or 1003 in our metadata store - labsdbaccount.account_host on m5-master.

Details

Related Gerrit Patches:

Event Timeline

madhuvishy triaged this task as High priority.Dec 15 2017, 8:06 PM
madhuvishy created this task.

Change 398533 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] maintain-dbusers: Stop managing account creation for labsdb1001 and 1003

https://gerrit.wikimedia.org/r/398533

Change 398533 merged by Madhuvishy:
[operations/puppet@production] maintain-dbusers: Stop managing account creation for labsdb1001 and 1003

https://gerrit.wikimedia.org/r/398533

Marostegui added a subscriber: Marostegui.

I dropped all the status=absent accounts from labsdbaccount.account_host for labsdb1001 and 1003.

madhuvishy lowered the priority of this task from High to Medium.Dec 15 2017, 8:29 PM

The script is unbroken now and runs alright. Leaving this open until we remove all the metadata for labsdb1001 and 3 from labsdbaccount.account_host post decommission.

bd808 moved this task from Backlog to Wiki replicas on the Data-Services board.Jan 18 2018, 7:02 PM

What is the state of this now due to T184832?

What is the state of this now due to T184832?

I believe with https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/labs/db/maintain_dbusers.pp this is all sorted out. I'm assigning to @madhuvishy for her to confirm and resolve.

madhuvishy added a comment.EditedFeb 6 2018, 12:31 AM

@jcrespo I'd like to drop all the accounts metadata for labsdb1001 & 3 from labsdbaccounts.account_host on m5-master to close this task.

That would mean running:

delete from labsdbaccounts.account_host where hostname='labsdb1001.eqiad.wmnet';
delete from labsdbaccounts.account_host where hostname='labsdb1003.eqiad.wmnet';

I'm happy to back up the data first before running this. Does this sound good to you?

@madhuvishy data management of m* databases is something I do not handle or you do not have to ask permission for. My suggestion is to connect as the user of the service, and not as root (as to minimize mistakes), and just run those commands. We already perform backups weekly, but apparently this new db was not added. This should be a blocker of your maintenance there.

This is the new list of m5 databases being backed up- if you confirm that is as intended, you can proceed now we have proper[sic] backups.

root@dbstore2001:/srv/backups/m5.20180214102352$ ls *-schema-create.sql.gz
ceilometer-schema-create.sql.gz
designate_pool_manager-schema-create.sql.gz
designate-schema-create.sql.gz
glance-schema-create.sql.gz
keystone-schema-create.sql.gz
labsdbaccounts-schema-create.sql.gz
labspuppet-schema-create.sql.gz
neutron-schema-create.sql.gz
nodepooldb-schema-create.sql.gz
nova-schema-create.sql.gz
striker-schema-create.sql.gz

This is the new list of m5 databases being backed up- if you confirm that is as intended, you can proceed now we have proper[sic] backups.

root@dbstore2001:/srv/backups/m5.20180214102352$ ls *-schema-create.sql.gz
ceilometer-schema-create.sql.gz
designate_pool_manager-schema-create.sql.gz
designate-schema-create.sql.gz
glance-schema-create.sql.gz
keystone-schema-create.sql.gz
labsdbaccounts-schema-create.sql.gz
labspuppet-schema-create.sql.gz
neutron-schema-create.sql.gz
nodepooldb-schema-create.sql.gz
nova-schema-create.sql.gz
striker-schema-create.sql.gz

@jcrespo That looks good, thank you!

I've dropped the metadata for labsdb1001 and 1003 from labsdbaccounts.account_host. It now looks like

madhuvishy@labstore1004:~$ mysql -h m5-master.eqiad.wmnet -u labsdbaccounts -p -e "select hostname, count(*) from labsdbaccounts.account_host group by hostname;"
+------------------------+----------+
| hostname               | count(*) |
+------------------------+----------+
| labsdb1005.eqiad.wmnet |     3115 |
| labsdb1009.eqiad.wmnet |     3114 |
| labsdb1010.eqiad.wmnet |     3114 |
| labsdb1011.eqiad.wmnet |     3114 |
+------------------------+----------+
madhuvishy closed this task as Resolved.Feb 14 2018, 6:02 PM