In order to decom db1048, we need to setup a newer host with that role. We can use db1059, which is "easy" to remove from s4, and pool it as the new m3 replica.
db1059 is now in production usage, only task left is to get rid of db1048.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Duplicate | • Cmjohnson | T160731 Decom db1048 (BBU Faulty - slave lagging) | |||
| Resolved | None | T134476 Decommission old coredb machines (<=db1050) | |||
| Resolved | • Cmjohnson | T175679 Decommission db1048 (was Move m3 slave to db1059) | |||
| Resolved | jcrespo | T162593 Run pt-table-checksum on s4 (commonswiki) |
Event Timeline
Comment Actions
Change 377455 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1059, pool db1097 as api with low load
Comment Actions
Change 377455 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1059, pool db1097 as api with low load
Comment Actions
Change 377468 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Remove all references of db1059 from mediawiki
Comment Actions
Change 377468 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Remove all references of db1059 from mediawiki
Comment Actions
Change 377474 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Move db1059 from mediawiki to misc (m3)
Comment Actions
Change 377474 merged by Jcrespo:
[operations/puppet@production] mariadb: Move db1059 from mediawiki to misc (m3)
Comment Actions
Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:
['db1059.eqiad.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201709121603_jynus_3476.log.
Comment Actions
Change 377687 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Repoint m3 secondary host (replica) to db1059
Comment Actions
Change 377687 merged by Jcrespo:
[operations/puppet@production] mariadb: Repoint m3 secondary host (replica) to db1059
Comment Actions
Change 377693 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] phabricator/mariadb: Update database configuration for stretch/10.1
Comment Actions
@mmodell We have to upgrade the hardware for phabricator databases. What do you think of doing also this thursday a master switchover and upgrade to stretch/mariadb 10.1, enable TLS and setup the firewall. It should be a few seconds of restarting phabricator to get the new connections, if something goes bad, we revert to the current server.
Comment Actions
Change 377701 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] misc dbs: Repoint m3-slave to the new replica server db1059
Comment Actions
Change 377701 merged by Jcrespo:
[operations/dns@master] misc dbs: Repoint m3-slave to the new replica server db1059
Comment Actions
Change 377705 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] decommission: Set db1048 with a spare role
Comment Actions
Change 377705 merged by Jcrespo:
[operations/puppet@production] decommission: Set db1048 with a spare role
Comment Actions
db1048 is now ready to be decommissioned, it is set as spare, but it still needs to be fully deleted from the configuration and infrastructure (installer, site.pp).
Comment Actions
@jcrespo Any time will work for me, there is scheduled maintenance at midnight tonight (UTC) but if it's just a few seconds of downtime I think we can do it whenever.
Comment Actions
Let's wait a bit more. I may have to talk to you abut setting up TLS for php and changing passwords, let's talk and aim for next week (but we shouldn't delay it much).
Comment Actions
@mmodell This is still needed, but this and the next week are going to be problematic. As a heads up, we may need to merge some puppet changes simultneously on phabricator database and all its application servers. I will try to send you a calendar proposal at some point.
Comment Actions
@Cmjohnson sorry for the confusion- indeed it is ok to put down db1048. All other conversations were about failover to db1059 to substitute db1043 (this cannot be done yet). We will handle that on a separate ticket.
Comment Actions
All non-interruptible steps have been completed. Still needs wiping/removal from rack
Comment Actions
Change 377693 merged by Jcrespo:
[operations/puppet@production] phabricator/mariadb: Update database configuration for stretch/10.1