Page MenuHomePhabricator
Create Task
Maniphest T186320

Decommission db1051-db1060 (DBA tracking)
Closed, ResolvedPublic
Actions

Description

As part of the regular refresh, and given the aging database hosts, plan the purchase, setup of new databases and the removal of older ones.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+4 -4mariadb: Switchover m3-master to db1072
operations/puppetproduction+10 -8mariadb: Failover m3-master to db1072
operations/dnsmaster+2 -2mariadb: Update misc replica CNAME for m2 and m3
operations/softwaremaster+1 -1mariadb: Remove db1051, to be decommissioned, add db1065
operations/puppetproduction+16 -7mariadb: Switchover m2-master to db1065
operations/puppetproduction+8 -8mariadb: Failover m2-master to db1065
operations/dnsmaster+4 -4mariadb: Remove old references to db105* and codfw hosts at dns
operations/mediawiki-configmaster+1 -1mariadb: Pool db1076 back with full weight
operations/mediawiki-configmaster+2 -2mariadb: Pool db1076 with low load, increase db1066 load
operations/puppetproduction+1 -1install_server: Revert db recipe for all databases
operations/puppetproduction+1 -3mariadb: Allow reimage of db107* hosts
operations/mediawiki-configmaster+1 -1mariadb: Depool db1076 for maintenance
operations/mediawiki-configmaster+2 -2mariadb: Pool db1066 with low load after reimage
operations/puppetproduction+1 -2mariadb: Productionize db1066 after reimage
operations/puppetproduction+7 -7mariadb: Move db1066 from s1 to an s2 master candidate
operations/softwaremaster+1 -1dbhosts: Move db1066 from s1 to s2
operations/mediawiki-configmaster+8 -12mariadb: Fully pool db1123, remove db1072
operations/puppetproduction+6 -5mariadb: Prepare db1072 for stretch reimage and move it to m3
operations/softwaremaster+1 -1x1.hosts: db1069 is the new x1 master
operations/mediawiki-configmaster+2 -2db-eqiad.php: Promote db1069 to be x1 master
operations/puppetproduction+3 -3mariadb: db1069 is now x1 master
operations/mediawiki-configmaster+14 -13mariadb: Pool new vslow,dump host on s4 (db1121), move db1064 to x1
operations/mediawiki-configmaster+1 -1mariadb: Fully pool back db1056 and db1069 as x1 replicas
operations/mediawiki-configmaster+3 -3mariadb: Repool db1056 and db1069 with low load after maintenance
operations/mediawiki-configmaster+2 -2mariadb: Depool db1056
operations/softwaremaster+1 -1mariadb: Move db1069 from s7 to x1
operations/mediawiki-configmaster+1 -1mariadb: Move db1069 from s7 to x1 (while still fully depooled)
operations/puppetproduction+9 -7mariadb: Move db1069 from s7 to x1 and enable its reimage
operations/mediawiki-configmaster+6 -10mariadb: Depool db1060, pool fully db1090 instances
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a subscriber: ops-monitoring-bot.Apr 30 2018, 1:50 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db1069.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804301349_jynus_6466.log.

ops-monitoring-bot added a comment.Apr 30 2018, 2:10 PM

Completed auto-reimage of hosts:

['db1069.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Apr 30 2018, 2:20 PM

Change 429800 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Move db1069 from s7 to x1 (while still fully depooled)

https://gerrit.wikimedia.org/r/429800

gerritbot added a comment.Apr 30 2018, 2:22 PM

Change 429805 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1056

https://gerrit.wikimedia.org/r/429805

gerritbot added a comment.Apr 30 2018, 2:25 PM

Change 429812 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] mariadb: Move db1069 from s7 to x1

https://gerrit.wikimedia.org/r/429812

gerritbot added a comment.Apr 30 2018, 2:26 PM

Change 429812 merged by Jcrespo:
[operations/software@master] mariadb: Move db1069 from s7 to x1

https://gerrit.wikimedia.org/r/429812

gerritbot added a comment.Apr 30 2018, 3:25 PM

Change 429824 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1056 and db1069 with low load after maintenance

https://gerrit.wikimedia.org/r/429824

gerritbot added a comment.Apr 30 2018, 3:29 PM

Change 429824 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool db1056 and db1069 with low load after maintenance

https://gerrit.wikimedia.org/r/429824

gerritbot added a comment.Apr 30 2018, 3:30 PM

Change 429826 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Fully pool back db1056 and db1069 as x1 replicas

https://gerrit.wikimedia.org/r/429826

gerritbot added a comment.Apr 30 2018, 3:45 PM

Change 429826 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Fully pool back db1056 and db1069 as x1 replicas

https://gerrit.wikimedia.org/r/429826

gerritbot added a comment.May 2 2018, 2:00 PM

Change 430377 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool new vslow,dump host on s4 (db1121), move db1064 to x1

https://gerrit.wikimedia.org/r/430377

gerritbot added a comment.May 2 2018, 2:28 PM

Change 430377 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Pool new vslow,dump host on s4 (db1121), move db1064 to x1

https://gerrit.wikimedia.org/r/430377

jcrespo removed a subtask: Unknown Object (Task).May 4 2018, 11:54 AM
gerritbot added a comment.May 7 2018, 1:37 PM

Change 431566 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Promote db1069 to be x1 master

https://gerrit.wikimedia.org/r/431566

gerritbot added a comment.May 7 2018, 1:39 PM

Change 431567 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] x1.hosts: db1069 is the new x1 master

https://gerrit.wikimedia.org/r/431567

gerritbot added a comment.May 7 2018, 1:41 PM

Change 431568 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db1069 is now x1 master

https://gerrit.wikimedia.org/r/431568

Stashbot added a subscriber: Stashbot.May 8 2018, 5:28 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-08T05:28:22Z] <marostegui> Disable gtid on db1069 an db2034 before x1 failover - T186320

Stashbot added a comment.May 8 2018, 5:29 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-08T05:29:48Z] <marostegui> Disable puppet on db1055 and db1069 before x1 failover - T186320

Stashbot added a comment.May 8 2018, 5:36 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-08T05:36:35Z] <marostegui> Move dbstore1002:x1 under db1069 for x1 failover - T186320

Stashbot added a comment.May 8 2018, 5:41 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-08T05:41:22Z] <marostegui> Move db2034 under db1069 for x1 failover - T186320

gerritbot added a comment.May 8 2018, 5:42 AM

Change 431568 merged by Marostegui:
[operations/puppet@production] mariadb: db1069 is now x1 master

https://gerrit.wikimedia.org/r/431568

gerritbot added a comment.May 8 2018, 6:00 AM

Change 431566 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Promote db1069 to be x1 master

https://gerrit.wikimedia.org/r/431566

gerritbot added a comment.May 8 2018, 6:11 AM

Change 431567 merged by jenkins-bot:
[operations/software@master] x1.hosts: db1069 is the new x1 master

https://gerrit.wikimedia.org/r/431567

gerritbot added a comment.May 10 2018, 11:01 AM

Change 432358 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Prepare db1072 for stretch reimage and move it to m3

https://gerrit.wikimedia.org/r/432358

gerritbot added a comment.May 10 2018, 2:32 PM

Change 432358 merged by Jcrespo:
[operations/puppet@production] mariadb: Prepare db1072 for stretch reimage and move it to m3

https://gerrit.wikimedia.org/r/432358

ops-monitoring-bot added a comment.May 10 2018, 3:39 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db1072.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805101539_jynus_4853.log.

ops-monitoring-bot added a comment.May 10 2018, 4:00 PM

Completed auto-reimage of hosts:

['db1072.eqiad.wmnet']

and were ALL successful.

jcrespo added a comment.May 10 2018, 5:11 PM

db1072 is loading m3 from the last logical backup.

gerritbot added a comment.May 10 2018, 5:47 PM

Change 432409 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Fully pool db1123, remove db1072

https://gerrit.wikimedia.org/r/432409

gerritbot added a comment.May 10 2018, 5:55 PM

Change 432409 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Fully pool db1123, remove db1072

https://gerrit.wikimedia.org/r/432409

gerritbot added a comment.May 11 2018, 8:20 AM

Change 432552 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Move db1066 from s1 to an s2 master candidate

https://gerrit.wikimedia.org/r/432552

gerritbot added a comment.May 11 2018, 8:24 AM

Change 432554 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dbhosts: Move db1066 from s1 to s2

https://gerrit.wikimedia.org/r/432554

gerritbot added a comment.May 11 2018, 8:26 AM

Change 432554 merged by Jcrespo:
[operations/software@master] dbhosts: Move db1066 from s1 to s2

https://gerrit.wikimedia.org/r/432554

gerritbot added a comment.May 11 2018, 9:14 AM

Change 432552 merged by Jcrespo:
[operations/puppet@production] mariadb: Move db1066 from s1 to an s2 master candidate

https://gerrit.wikimedia.org/r/432552

ops-monitoring-bot added a comment.May 11 2018, 9:17 AM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db1066.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805110916_jynus_29532.log.

gerritbot added a comment.May 11 2018, 9:20 AM

Change 432560 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1076 for maintenance

https://gerrit.wikimedia.org/r/432560

gerritbot added a comment.May 11 2018, 9:23 AM

Change 432560 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1076 for maintenance

https://gerrit.wikimedia.org/r/432560

gerritbot added a comment.May 11 2018, 11:36 AM

Change 432565 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Productionize db1066 after reimage

https://gerrit.wikimedia.org/r/432565

gerritbot added a comment.May 11 2018, 11:39 AM

Change 432566 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool db1066 with low load after reimage

https://gerrit.wikimedia.org/r/432566

gerritbot added a comment.May 11 2018, 11:42 AM

Change 432565 merged by Jcrespo:
[operations/puppet@production] mariadb: Productionize db1066 after reimage

https://gerrit.wikimedia.org/r/432565

gerritbot added a comment.May 11 2018, 12:04 PM

Change 432566 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Pool db1066 with low load after reimage

https://gerrit.wikimedia.org/r/432566

gerritbot added a comment.May 11 2018, 12:55 PM

Change 432575 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Allow reimage of db107* hosts

https://gerrit.wikimedia.org/r/432575

gerritbot added a comment.May 11 2018, 12:59 PM

Change 432575 merged by Jcrespo:
[operations/puppet@production] mariadb: Allow reimage of db107* hosts

https://gerrit.wikimedia.org/r/432575

gerritbot added a comment.May 11 2018, 1:03 PM

Change 432577 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] install_server: Revert db recipe for all databases

https://gerrit.wikimedia.org/r/432577

ops-monitoring-bot added a comment.May 11 2018, 1:05 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db1076.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805111305_jynus_8054.log.

ops-monitoring-bot added a comment.May 11 2018, 1:26 PM

Completed auto-reimage of hosts:

['db1076.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.May 11 2018, 1:29 PM

Change 432577 merged by Jcrespo:
[operations/puppet@production] install_server: Revert db recipe for all databases

https://gerrit.wikimedia.org/r/432577

gerritbot added a comment.May 11 2018, 1:52 PM

Change 432581 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool db1076 with low load, increase db1066 load

https://gerrit.wikimedia.org/r/432581

gerritbot added a comment.May 11 2018, 2:01 PM

Change 432581 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Pool db1076 with low load, increase db1066 load

https://gerrit.wikimedia.org/r/432581

gerritbot added a comment.May 11 2018, 3:58 PM

Change 432602 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool db1076 back with full weight

https://gerrit.wikimedia.org/r/432602

gerritbot added a comment.May 11 2018, 5:17 PM

Change 432602 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Pool db1076 back with full weight

https://gerrit.wikimedia.org/r/432602

Marostegui added a subtask: T194867: BBU issues on db1054 (s2 primary master).May 17 2018, 8:29 AM
Marostegui mentioned this in T194870: Failover s2 primary master.May 17 2018, 8:47 AM
Marostegui mentioned this in T194867: BBU issues on db1054 (s2 primary master).May 21 2018, 6:04 AM
Marostegui closed subtask T194867: BBU issues on db1054 (s2 primary master) as Declined.
gerritbot added a comment.May 24 2018, 2:11 PM

Change 434920 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] mariadb: Remove old references to db105* hosts at dns

https://gerrit.wikimedia.org/r/434920

gerritbot added a comment.May 30 2018, 8:18 AM

Change 434920 merged by Jcrespo:
[operations/dns@master] mariadb: Remove old references to db105* and codfw hosts at dns

https://gerrit.wikimedia.org/r/434920

gerritbot added a comment.Jun 6 2018, 9:55 AM

Change 437703 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Failover m2-master to db1065

https://gerrit.wikimedia.org/r/437703

gerritbot added a comment.Jun 6 2018, 9:58 AM

Change 437707 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Failover m3-master to db1072

https://gerrit.wikimedia.org/r/437707

gerritbot added a comment.Jun 6 2018, 10:07 AM

Change 437710 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] mariadb: Update misc replica CNAME for m2 and m3

https://gerrit.wikimedia.org/r/437710

gerritbot added a comment.Jun 6 2018, 10:24 AM

Change 437714 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Switchover m2-master to db1065

https://gerrit.wikimedia.org/r/437714

gerritbot added a comment.Jun 6 2018, 10:24 AM

Change 437715 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Switchover m3-master to db1072

https://gerrit.wikimedia.org/r/437715

gerritbot added a comment.Jun 6 2018, 2:05 PM

Change 437703 merged by Jcrespo:
[operations/puppet@production] mariadb: Failover m2-master to db1065

https://gerrit.wikimedia.org/r/437703

gerritbot added a comment.Jun 6 2018, 2:17 PM

Change 437714 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Switchover m2-master to db1065

https://gerrit.wikimedia.org/r/437714

gerritbot added a comment.Jun 6 2018, 2:18 PM

Change 437714 merged by Jcrespo:
[operations/puppet@production] mariadb: Switchover m2-master to db1065

https://gerrit.wikimedia.org/r/437714

gerritbot added a comment.Jun 6 2018, 2:57 PM

Change 437769 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] mariadb: Remove db1051, to be decommissioned, add db1065

https://gerrit.wikimedia.org/r/437769

gerritbot added a comment.Jun 6 2018, 3:17 PM

Change 437769 merged by Jcrespo:
[operations/software@master] mariadb: Remove db1051, to be decommissioned, add db1065

https://gerrit.wikimedia.org/r/437769

gerritbot added a comment.Jun 6 2018, 4:50 PM

Change 437710 merged by Jcrespo:
[operations/dns@master] mariadb: Update misc replica CNAME for m2 and m3

https://gerrit.wikimedia.org/r/437710

gerritbot added a comment.Jun 7 2018, 5:47 AM

Change 437707 merged by Marostegui:
[operations/puppet@production] mariadb: Failover m3-master to db1072

https://gerrit.wikimedia.org/r/437707

gerritbot added a comment.Jun 7 2018, 5:51 AM

Change 437715 merged by Marostegui:
[operations/puppet@production] mariadb: Switchover m3-master to db1072

https://gerrit.wikimedia.org/r/437715

Marostegui added a comment.EditedJun 8 2018, 3:45 PM

To sum up the pending active hosts:

db1054 (s2 primary master): Failover scheduled T194870
db1052 (s1 primary master): Failover scheduled T197069

Marostegui added a subtask: T197063: Decommission db1054.Jun 13 2018, 6:38 AM
Marostegui closed subtask T194870: Failover s2 primary master as Resolved.
Marostegui mentioned this in T197069: Failover db1052 (s1) db primary master.Jun 13 2018, 8:22 AM
Marostegui added a subtask: T197069: Failover db1052 (s1) db primary master.
Marostegui mentioned this in T197134: Announce 30 minutes read-only time for enwiki 18th July 06:00AM UTC.Jun 13 2018, 3:40 PM
jcrespo added a comment.Jun 13 2018, 4:51 PM

db1054 is pending wait everything is working as expected on s2- db1052 process has not started yet. All others are ready for robh/dcops to continue as noted on the subtasks.

Marostegui added a comment.Jun 26 2018, 6:03 AM

db1054 is now handed over to DCOps for decommissioning.
The only pending host is db1052 (s1 primary master) which is scheduled to be failed over the 18th July (T197069)

• Vvjjkkii reopened subtask T194867: BBU issues on db1054 (s2 primary master) as Open.Jul 1 2018, 1:09 AM
• Vvjjkkii reopened subtask T194870: Failover s2 primary master as Open.
Wong128hk closed subtask T194867: BBU issues on db1054 (s2 primary master) as Declined.Jul 1 2018, 2:28 AM
Marostegui closed subtask T194870: Failover s2 primary master as Resolved.Jul 2 2018, 5:20 AM
Marostegui closed subtask T197069: Failover db1052 (s1) db primary master as Resolved.Jul 18 2018, 6:43 AM
Marostegui moved this task from In progress to Done on the DBA board.Aug 1 2018, 7:24 AM

This is essentially all done - all the hosts have been handed over to DCOps for the last steps of the decommissioning process.

Cmjohnson closed subtask T195484: Decommission db1051 as Resolved.Aug 7 2018, 5:04 PM
Cmjohnson closed subtask T194634: Decommission db1053 as Resolved.
Cmjohnson closed subtask T197063: Decommission db1054 as Resolved.
Cmjohnson closed subtask T194118: Decommission db1055 as Resolved.Aug 7 2018, 5:06 PM
Cmjohnson closed subtask T193736: Decommission db1056 as Resolved.
Cmjohnson closed subtask T196606: Decommission db1059 as Resolved.
Cmjohnson closed subtask T193732: Decommission db1060 as Resolved.
Cmjohnson closed subtask T199861: Decommission db1052 as Resolved.Aug 21 2018, 5:05 PM
Marostegui closed this task as Resolved.Aug 21 2018, 5:37 PM
Marostegui added subscribers: Cmjohnson, RobH.

All these hosts have now been fully decommissioned
Thanks @RobH a @Cmjohnson for all the help!

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL