Page MenuHomePhabricator

Decommission old coredb machines (<=db2042)
Closed, ResolvedPublic

Description

The following hosts are scheduled to be replaced by 10th July 2019.
They need to be handed over to DCOPs once they are removed from service

This list tracks the servers that have been sent over to DCOPs for retirement and its corresponding task.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 514224 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Move db2058 to s6

https://gerrit.wikimedia.org/r/514224

Change 514225 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2058 to s6

https://gerrit.wikimedia.org/r/514225

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2058.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906040729_marostegui_118594.log.

Completed auto-reimage of hosts:

['db2058.codfw.wmnet']

and were ALL successful.

Change 514232 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2046

https://gerrit.wikimedia.org/r/514232

Change 514232 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2046

https://gerrit.wikimedia.org/r/514232

Mentioned in SAL (#wikimedia-operations) [2019-06-04T08:04:21Z] <marostegui> Stop MySQL on db2046 to clone db2058 - T221533

Mentioned in SAL (#wikimedia-operations) [2019-06-05T05:22:49Z] <marostegui> Change replication topology on m3 codfw to promote db2065 as codfw master instead of db2042 - T221533

Marostegui updated the task description. (Show Details)Jun 5 2019, 1:08 PM
Marostegui updated the task description. (Show Details)Jun 6 2019, 5:33 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-06T05:41:46Z] <marostegui> Upgrade MySQL on s6 codfw hosts in preparation for s6 codfw master failover - T221533

Mentioned in SAL (#wikimedia-operations) [2019-06-06T06:31:14Z] <marostegui> Start topology changes on s6 codfw to promote db2046 as master - T221533

Change 514657 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2046 to s6 codfw master

https://gerrit.wikimedia.org/r/514657

Change 514658 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Promote db2046 to s6 codfw master

https://gerrit.wikimedia.org/r/514658

Change 514657 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2046 to s6 codfw master

https://gerrit.wikimedia.org/r/514657

Change 514658 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Promote db2046 to s6 codfw master

https://gerrit.wikimedia.org/r/514658

Mentioned in SAL (#wikimedia-operations) [2019-06-06T07:16:02Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2046 to s6 master as db2039 will be decommissioned T221533 (duration: 00m 55s)

Change 515003 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db1132, enable notifications

https://gerrit.wikimedia.org/r/515003

Change 515003 merged by Marostegui:
[operations/puppet@production] mariadb: db1132, enable notifications

https://gerrit.wikimedia.org/r/515003

Change 515004 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Move db2051 from s4 to s2

https://gerrit.wikimedia.org/r/515004

Change 515004 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Move db2051 from s4 to s2

https://gerrit.wikimedia.org/r/515004

Change 515005 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reimage db2051

https://gerrit.wikimedia.org/r/515005

Change 515005 merged by Marostegui:
[operations/puppet@production] install_server: Allow reimage db2051

https://gerrit.wikimedia.org/r/515005

Change 515006 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db2051 from s4 to s2

https://gerrit.wikimedia.org/r/515006

Change 515006 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2051 from s4 to s2

https://gerrit.wikimedia.org/r/515006

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2051.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906070606_marostegui_29774.log.

Completed auto-reimage of hosts:

['db2051.codfw.wmnet']

Of which those FAILED:

['db2051.codfw.wmnet']

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2051.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906070640_marostegui_35129.log.

Completed auto-reimage of hosts:

['db2051.codfw.wmnet']

Of which those FAILED:

['db2051.codfw.wmnet']

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2051.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906070640_marostegui_35168.log.

Completed auto-reimage of hosts:

['db2051.codfw.wmnet']

Of which those FAILED:

['db2051.codfw.wmnet']

Mentioned in SAL (#wikimedia-operations) [2019-06-07T08:24:08Z] <marostegui> Upgrade s2 codfw to 10.1.39 in preparation for its codfw failover - T221533

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2051.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906070943_marostegui_70075.log.

Completed auto-reimage of hosts:

['db2051.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-06-17T05:16:36Z] <marostegui> Stop MySQL on db2107 to clone db2051 - T221533

Change 517586 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2051: Enable notifications

https://gerrit.wikimedia.org/r/517586

Change 517586 merged by Marostegui:
[operations/puppet@production] db2051: Enable notifications

https://gerrit.wikimedia.org/r/517586

Change 517591 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2039

https://gerrit.wikimedia.org/r/517591

Change 517591 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2039

https://gerrit.wikimedia.org/r/517591

Mentioned in SAL (#wikimedia-operations) [2019-06-18T07:03:50Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2039 from config T221533 (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2019-06-18T07:04:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2039 from config T221533 (duration: 00m 46s)

Marostegui updated the task description. (Show Details)Jun 18 2019, 8:26 AM

Change 518193 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Pool db2051 into s2

https://gerrit.wikimedia.org/r/518193

Change 518193 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Pool db2051 into s2

https://gerrit.wikimedia.org/r/518193

Marostegui mentioned this in Unknown Object (Task).Jun 24 2019, 2:43 PM
Dzahn removed a subscriber: Dzahn.Jun 27 2019, 10:32 PM
Marostegui mentioned this in Unknown Object (Task).Jul 2 2019, 5:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-08T06:36:17Z] <marostegui> Run compare for s5 main tables on db2038 vs db2059 - T221533

Marostegui updated the task description. (Show Details)Jul 9 2019, 8:47 AM
Marostegui updated the task description. (Show Details)

Change 525799 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2038, pool db2113

https://gerrit.wikimedia.org/r/525799

Change 525799 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2038, pool db2113

https://gerrit.wikimedia.org/r/525799

Mentioned in SAL (#wikimedia-operations) [2019-07-26T12:35:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db2123 into s5 vslow T221533 (duration: 00m 50s)

Mentioned in SAL (#wikimedia-operations) [2019-07-29T07:29:32Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2038 from config T221533 (duration: 00m 50s)

Mentioned in SAL (#wikimedia-operations) [2019-07-29T07:30:41Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2038 from config T221533 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-07-29T12:44:49Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db2128 into s5 api T221533 (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2019-07-29T12:45:45Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Provision db2128 into s5 api T221533 (duration: 00m 47s)

Marostegui updated the task description. (Show Details)Jul 30 2019, 8:21 AM

Change 528070 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2107 to codfw s2 master

https://gerrit.wikimedia.org/r/528070

Mentioned in SAL (#wikimedia-operations) [2019-08-05T08:21:27Z] <marostegui> Switchover s2 codfw master from db2035 to db2107 - T221533 T220170

Change 528070 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2107 to codfw s2 master

https://gerrit.wikimedia.org/r/528070

Marostegui updated the task description. (Show Details)Aug 5 2019, 8:45 AM
Marostegui closed this task as Resolved.Aug 8 2019, 5:57 AM
Marostegui updated the task description. (Show Details)

All these hosts have been sent to DC-Ops for the last steps of their decommissioning process, so resolving this.