Page MenuHomePhabricator
Create Task
Maniphest T221533

Decommission old coredb machines (<=db2042)
Closed, ResolvedPublic
Actions

Description

The following hosts are scheduled to be replaced by 10th July 2019.
They need to be handed over to DCOPs once they are removed from service

This list tracks the servers that have been sent over to DCOPs for retirement and its corresponding task.

Details

SubjectRepoBranchLines +/-
mariadb: Promote db2107 to codfw s2 masteroperations/puppetproduction+3 -3
db-codfw.php: Depool db2038, pool db2113operations/mediawiki-configmaster+4 -4
db-codfw.php: Pool db2051 into s2operations/mediawiki-configmaster+1 -1
db-eqiad,db-codfw.php: Remove db2039operations/mediawiki-configmaster+0 -3
db2051: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Move db2051 from s4 to s2operations/puppetproduction+4 -4
install_server: Allow reimage db2051operations/puppetproduction+1 -1
db-codfw.php: Move db2051 from s4 to s2operations/mediawiki-configmaster+1 -1
mariadb: db1132, enable notificationsoperations/puppetproduction+1 -1
db-codfw.php: Promote db2046 to s6 codfw masteroperations/mediawiki-configmaster+3 -3
mariadb: Promote db2046 to s6 codfw masteroperations/puppetproduction+5 -5
db-codfw.php: Depool db2046operations/mediawiki-configmaster+1 -1
mariadb: Move db2058 to s6operations/puppetproduction+6 -5
db-codfw.php: Move db2058 to s6operations/mediawiki-configmaster+1 -1
mariadb: Move db2065 to m3operations/puppetproduction+9 -4
db-eqiad,db-codfw.php: Remove db2037 from configoperations/mediawiki-configmaster+0 -3
db2065: Disable notificationsoperations/puppetproduction+1 -0
db-eqiad,db-codfw.php: Remove db2065operations/mediawiki-configmaster+0 -3
db2070: Enable notificationsoperations/puppetproduction+1 -1
db-codfw.php: Promote db2070 to m5 codfw masteroperations/mediawiki-configmaster+2 -2
mariadb: Move db2070 from s1 to m5operations/puppetproduction+5 -5
db-codfw.php: Move db2070 to m5operations/mediawiki-configmaster+1 -1
db-eqiad,db-codfw.php: Remove db2036operations/mediawiki-configmaster+7 -15
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jun 4 2019, 7:15 AM

Change 514224 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Move db2058 to s6

https://gerrit.wikimedia.org/r/514224

gerritbot added a comment.Jun 4 2019, 7:21 AM

Change 514225 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2058 to s6

https://gerrit.wikimedia.org/r/514225

ops-monitoring-bot added a comment.Jun 4 2019, 7:29 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2058.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906040729_marostegui_118594.log.

ops-monitoring-bot added a comment.Jun 4 2019, 7:56 AM

Completed auto-reimage of hosts:

['db2058.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Jun 4 2019, 7:59 AM

Change 514232 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2046

https://gerrit.wikimedia.org/r/514232

gerritbot added a comment.Jun 4 2019, 8:03 AM

Change 514232 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2046

https://gerrit.wikimedia.org/r/514232

Stashbot added a comment.Jun 4 2019, 8:04 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-04T08:04:21Z] <marostegui> Stop MySQL on db2046 to clone db2058 - T221533

Maintenance_bot removed a project: Patch-For-Review.Jun 4 2019, 8:10 AM
Stashbot added a comment.Jun 5 2019, 5:22 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-05T05:22:49Z] <marostegui> Change replication topology on m3 codfw to promote db2065 as codfw master instead of db2042 - T221533

Marostegui updated the task description. (Show Details)Jun 5 2019, 1:08 PM
Marostegui updated the task description. (Show Details)Jun 6 2019, 5:33 AM
Stashbot added a comment.Jun 6 2019, 5:41 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-06T05:41:46Z] <marostegui> Upgrade MySQL on s6 codfw hosts in preparation for s6 codfw master failover - T221533

Stashbot added a comment.Jun 6 2019, 6:31 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-06T06:31:14Z] <marostegui> Start topology changes on s6 codfw to promote db2046 as master - T221533

gerritbot added a comment.Jun 6 2019, 7:07 AM

Change 514657 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2046 to s6 codfw master

https://gerrit.wikimedia.org/r/514657

gerritbot added a project: Patch-For-Review.Jun 6 2019, 7:07 AM
gerritbot added a comment.Jun 6 2019, 7:11 AM

Change 514658 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Promote db2046 to s6 codfw master

https://gerrit.wikimedia.org/r/514658

gerritbot added a comment.Jun 6 2019, 7:13 AM

Change 514657 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2046 to s6 codfw master

https://gerrit.wikimedia.org/r/514657

gerritbot added a comment.Jun 6 2019, 7:14 AM

Change 514658 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Promote db2046 to s6 codfw master

https://gerrit.wikimedia.org/r/514658

Stashbot added a comment.Jun 6 2019, 7:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-06T07:16:02Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2046 to s6 master as db2039 will be decommissioned T221533 (duration: 00m 55s)

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2019, 8:10 AM
gerritbot added a comment.Jun 7 2019, 5:09 AM

Change 515003 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db1132, enable notifications

https://gerrit.wikimedia.org/r/515003

gerritbot added a project: Patch-For-Review.Jun 7 2019, 5:09 AM

Change 515003 merged by Marostegui:
[operations/puppet@production] mariadb: db1132, enable notifications

https://gerrit.wikimedia.org/r/515003

gerritbot added a comment.Jun 7 2019, 5:37 AM

Change 515004 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Move db2051 from s4 to s2

https://gerrit.wikimedia.org/r/515004

gerritbot added a comment.Jun 7 2019, 5:39 AM

Change 515004 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Move db2051 from s4 to s2

https://gerrit.wikimedia.org/r/515004

gerritbot added a comment.Jun 7 2019, 5:46 AM

Change 515005 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reimage db2051

https://gerrit.wikimedia.org/r/515005

gerritbot added a comment.Jun 7 2019, 5:48 AM

Change 515005 merged by Marostegui:
[operations/puppet@production] install_server: Allow reimage db2051

https://gerrit.wikimedia.org/r/515005

gerritbot added a comment.Jun 7 2019, 5:56 AM

Change 515006 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db2051 from s4 to s2

https://gerrit.wikimedia.org/r/515006

gerritbot added a comment.Jun 7 2019, 6:04 AM

Change 515006 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2051 from s4 to s2

https://gerrit.wikimedia.org/r/515006

ops-monitoring-bot added a comment.Jun 7 2019, 6:06 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2051.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906070606_marostegui_29774.log.

Maintenance_bot removed a project: Patch-For-Review.Jun 7 2019, 6:10 AM
ops-monitoring-bot added a comment.Jun 7 2019, 6:35 AM

Completed auto-reimage of hosts:

['db2051.codfw.wmnet']

Of which those FAILED:

['db2051.codfw.wmnet']
ops-monitoring-bot added a comment.Jun 7 2019, 6:40 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2051.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906070640_marostegui_35129.log.

ops-monitoring-bot added a comment.Jun 7 2019, 6:40 AM

Completed auto-reimage of hosts:

['db2051.codfw.wmnet']

Of which those FAILED:

['db2051.codfw.wmnet']
ops-monitoring-bot added a comment.Jun 7 2019, 6:40 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2051.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906070640_marostegui_35168.log.

ops-monitoring-bot added a comment.Jun 7 2019, 6:55 AM

Completed auto-reimage of hosts:

['db2051.codfw.wmnet']

Of which those FAILED:

['db2051.codfw.wmnet']
Stashbot added a comment.Jun 7 2019, 8:24 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-07T08:24:08Z] <marostegui> Upgrade s2 codfw to 10.1.39 in preparation for its codfw failover - T221533

ops-monitoring-bot added a comment.Jun 7 2019, 9:43 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2051.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906070943_marostegui_70075.log.

ops-monitoring-bot added a comment.Jun 7 2019, 10:09 AM

Completed auto-reimage of hosts:

['db2051.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 17 2019, 5:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-17T05:16:36Z] <marostegui> Stop MySQL on db2107 to clone db2051 - T221533

gerritbot added a comment.Jun 18 2019, 4:33 AM

Change 517586 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2051: Enable notifications

https://gerrit.wikimedia.org/r/517586

gerritbot added a project: Patch-For-Review.Jun 18 2019, 4:33 AM

Change 517586 merged by Marostegui:
[operations/puppet@production] db2051: Enable notifications

https://gerrit.wikimedia.org/r/517586

Maintenance_bot removed a project: Patch-For-Review.Jun 18 2019, 5:10 AM
gerritbot added a comment.Jun 18 2019, 7:00 AM

Change 517591 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2039

https://gerrit.wikimedia.org/r/517591

gerritbot added a project: Patch-For-Review.Jun 18 2019, 7:00 AM

Change 517591 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2039

https://gerrit.wikimedia.org/r/517591

Stashbot added a comment.Jun 18 2019, 7:03 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-18T07:03:50Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2039 from config T221533 (duration: 00m 51s)

Stashbot added a comment.Jun 18 2019, 7:04 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-18T07:04:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2039 from config T221533 (duration: 00m 46s)

Marostegui added a subtask: T225988: decommission db2039.Jun 18 2019, 7:07 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 18 2019, 7:10 AM
Marostegui updated the task description. (Show Details)Jun 18 2019, 8:26 AM
jbond closed subtask T225278: Installation failing on late_command.sh as Resolved.Jun 19 2019, 2:26 PM
gerritbot added a comment.Jun 21 2019, 5:02 AM

Change 518193 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Pool db2051 into s2

https://gerrit.wikimedia.org/r/518193

gerritbot added a project: Patch-For-Review.Jun 21 2019, 5:02 AM
gerritbot added a comment.Jun 21 2019, 5:06 AM

Change 518193 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Pool db2051 into s2

https://gerrit.wikimedia.org/r/518193

Marostegui mentioned this in Unknown Object (Task).Jun 24 2019, 2:43 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 24 2019, 3:10 PM
Dzahn unsubscribed.Jun 27 2019, 10:32 PM
Marostegui mentioned this in Unknown Object (Task).Jul 2 2019, 5:06 AM
RobH mentioned this in T227113: rack/setup/install db21[21-30].codfw.wmnet.Jul 2 2019, 4:32 PM
Stashbot added a comment.Jul 8 2019, 6:36 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-08T06:36:17Z] <marostegui> Run compare for s5 main tables on db2038 vs db2059 - T221533

Marostegui updated the task description. (Show Details)Jul 9 2019, 8:47 AM
Marostegui updated the task description. (Show Details)
jcrespo mentioned this in T225713: CPU scaling governor audit.Jul 15 2019, 4:44 PM
Papaul closed subtask T225988: decommission db2039 as Resolved.Jul 18 2019, 11:13 PM
gerritbot added a comment.Jul 26 2019, 12:29 PM

Change 525799 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2038, pool db2113

https://gerrit.wikimedia.org/r/525799

gerritbot added a project: Patch-For-Review.Jul 26 2019, 12:29 PM
gerritbot added a comment.Jul 26 2019, 12:32 PM

Change 525799 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2038, pool db2113

https://gerrit.wikimedia.org/r/525799

Stashbot added a comment.Jul 26 2019, 12:35 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-26T12:35:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db2123 into s5 vslow T221533 (duration: 00m 50s)

Maintenance_bot removed a project: Patch-For-Review.Jul 26 2019, 1:10 PM
Stashbot added a comment.Jul 29 2019, 7:29 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-29T07:29:32Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2038 from config T221533 (duration: 00m 50s)

Stashbot added a comment.Jul 29 2019, 7:30 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-29T07:30:41Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2038 from config T221533 (duration: 00m 46s)

Stashbot added a comment.Jul 29 2019, 12:44 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-29T12:44:49Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db2128 into s5 api T221533 (duration: 00m 47s)

Stashbot added a comment.Jul 29 2019, 12:45 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-29T12:45:45Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Provision db2128 into s5 api T221533 (duration: 00m 47s)

Marostegui updated the task description. (Show Details)Jul 30 2019, 8:21 AM
Marostegui added a subtask: T227565: decommission db2038.Jul 30 2019, 8:24 AM
Papaul closed subtask T225090: Decommission db2042 as Resolved.Aug 1 2019, 1:20 PM
gerritbot added a comment.Aug 5 2019, 8:00 AM

Change 528070 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2107 to codfw s2 master

https://gerrit.wikimedia.org/r/528070

gerritbot added a project: Patch-For-Review.Aug 5 2019, 8:00 AM
Stashbot added a comment.Aug 5 2019, 8:21 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-05T08:21:27Z] <marostegui> Switchover s2 codfw master from db2035 to db2107 - T221533 T220170

Stashbot mentioned this in T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment.Aug 5 2019, 8:21 AM
gerritbot added a comment.Aug 5 2019, 8:27 AM

Change 528070 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2107 to codfw s2 master

https://gerrit.wikimedia.org/r/528070

Marostegui updated the task description. (Show Details)Aug 5 2019, 8:45 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 5 2019, 9:10 AM
Marostegui added a parent task: T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment.Aug 7 2019, 9:51 AM
Marostegui closed this task as Resolved.Aug 8 2019, 5:57 AM
Marostegui updated the task description. (Show Details)

All these hosts have been sent to DC-Ops for the last steps of their decommissioning process, so resolving this.

Papaul closed subtask T224720: Decommission db2037 as Resolved.Oct 1 2019, 12:31 AM
Papaul closed subtask T223885: Decommission db2036 as Resolved.Oct 1 2019, 12:39 AM
Papaul closed subtask T227565: decommission db2038 as Resolved.Oct 1 2019, 1:11 AM
Papaul closed subtask T224079: Decommission db2040 as Resolved.Oct 1 2019, 9:50 PM
Papaul closed subtask T223950: Decommission db2041 as Resolved.Oct 1 2019, 10:39 PM
Papaul closed subtask T229784: Decommission db2035 as Resolved.Oct 4 2019, 12:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits