Decommission db1015, db1035, db1044 and db1038
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	Oct 13 2016, 7:49 PM

Description

They are running out of space, and they are part of the easy-to-decom servers.

db1015 is ready: T173570
db1035 is ready: T176931
db1038 is ready: T177911
db1044 can be done at any time now. Pending puppet patches and https://gerrit.wikimedia.org/r/393747 (T181696)

Details

Subject	Repo	Branch	Lines +/-
mariadb: Decommission db1044	operations/mediawiki-config	master	+0 -2
mariadb: Disable all notifications on db1044, preparing to decom	operations/puppet	production	+1 -0
mariadb: Depool db1044	operations/mediawiki-config	master	+1 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T134476 Decommission old coredb machines (<=db1050)
Resolved	Marostegui	T148078 Decommission db1015, db1035, db1044 and db1038
Resolved	Marostegui	T164488 Run pt-table-checksum on s3
Resolved	• Cmjohnson	T177911 Decommission db1038

Event Timeline

jcrespo created this task.Oct 13 2016, 7:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 13 2016, 7:49 PM

jcrespo renamed this task from Decommission db1035 to Decommission db1015, db1035 and db1044.Oct 13 2016, 7:52 PM

jcrespo updated the task description. (Show Details)

Marostegui mentioned this in T153739: Defragment db1015.Dec 20 2016, 7:31 AM

High because they will complain of lack of space soon.

jcrespo renamed this task from Decommission db1015, db1035 and db1044 to Decommission db1015, db1035, db1044 and db1038.Feb 14 2017, 3:40 PM

jcrespo added a subtask: T154485: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038).

db1044 is db1095's master, so we need to look for another candidate within the shard (and change it to ROW) and make sure it has the same content as db1044 otherwise we will break db1095 :(

Have a look at my shard planning, I think I had some options there.

Yeah, you placed db1064 there as a master for it for s4.
We need to make sure they have the same data as otherwise ROW will not like that

s4 is... special in that regard.

worst case scenario we can move db1044's data to db1064 :-)

jcrespo closed subtask T154485: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) as Resolved.Apr 11 2017, 3:02 PM

jcrespo removed a subtask: T154485: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038).Apr 11 2017, 3:12 PM

Marostegui moved this task from Triage to Backlog on the DBA board.Apr 12 2017, 9:54 AM

jcrespo created subtask T164488: Run pt-table-checksum on s3.May 4 2017, 1:50 PM

I will try to run pt-table-checksum on the most important tables (revision, pagelinks, templatelinks, text) on s3 next week as db1015 is quite low on disk space already :-(

 root@db1015:~# df -hT /srv
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs   1.6T  1.5T   91G  95% /srv

db1015 was scheduled for decommission: T173570

Marostegui updated the task description. (Show Details)Sep 25 2017, 12:25 PM

db1035 scheduled for decommission: T176931

Marostegui closed subtask T164488: Run pt-table-checksum on s3 as Resolved.Oct 26 2017, 10:24 AM

Marostegui updated the task description. (Show Details)

Change 393740 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1044

https://gerrit.wikimedia.org/r/393740

gerritbot added a project: Patch-For-Review.Nov 28 2017, 10:38 AM

Change 393740 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1044

https://gerrit.wikimedia.org/r/393740

Change 393745 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Disable all notifications on db1044, preparing to decom

https://gerrit.wikimedia.org/r/393745

Change 393745 merged by Jcrespo:
[operations/puppet@production] mariadb: Disable all notifications on db1044, preparing to decom

https://gerrit.wikimedia.org/r/393745

Change 393747 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Decommission db1044

https://gerrit.wikimedia.org/r/393747

jcrespo assigned this task to Marostegui.Nov 28 2017, 2:54 PM

jcrespo updated the task description. (Show Details)

All hosts have now an individual decommission task - let's close this and follow up on each individual task

Change 393747 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Decommission db1044