Page MenuHomePhabricator

Productionize db213[2-5}
Closed, ResolvedPublic

Description

The following hosts are ready to go into misc services

  • db2132 A5 - m1
  • db2133 A6 - m2
  • db2134 B6 - m3
  • db2135 C6 - m5

Details

Related Gerrit Patches:
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Add db2135 to config
operations/puppet : productiondb2135: Enable notifications
operations/puppet : productionmariadb: Promote db2134 to m3 codfw master
operations/puppet : productiondb2134: Enable notifications
operations/puppet : productioninstall_server: Do not reimage db213[2-5]
operations/puppet : productionmariadb: Promote db2133 to m2 codfw master
operations/puppet : productiondb2133: Enable notifications
operations/puppet : productionmariadb: Promote db2132 to m1 codfw master
operations/puppet : productiondb2132: Enable notifications
operations/puppet : productionmariadb: Provision db2135 into m5
operations/puppet : productionmariadb: Provision db2134 into m3 codfw
operations/puppet : productionmariadb: Place db2133 into m2
operations/puppet : productioninstall_server: Reimage db213[3-5]
operations/puppet : productioninstall_server: Reimage db2132 with stretch
operations/puppet : productionmariadb: Pool db2132 into m1

Event Timeline

Marostegui triaged this task as Medium priority.Wed, Nov 13, 6:55 AM
Marostegui created this task.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)Wed, Nov 13, 7:15 AM

Change 550589 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Pool db2132 into m1

https://gerrit.wikimedia.org/r/550589

Change 550589 merged by Marostegui:
[operations/puppet@production] mariadb: Pool db2132 into m1

https://gerrit.wikimedia.org/r/550589

Change 550593 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2132 with stretch

https://gerrit.wikimedia.org/r/550593

Change 550593 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2132 with stretch

https://gerrit.wikimedia.org/r/550593

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2132.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911130800_marostegui_190803.log.

Completed auto-reimage of hosts:

['db2132.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-11-13T08:25:32Z] <marostegui> Stop MySQL on db2062 to copy its data to db2132 T238183

Marostegui added a subscriber: Papaul.

Change 550801 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db213[3-5]

https://gerrit.wikimedia.org/r/550801

Change 550801 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db213[3-5]

https://gerrit.wikimedia.org/r/550801

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2133.codfw.wmnet', 'db2134.codfw.wmnet', 'db2135.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911140823_marostegui_192468.log.

Completed auto-reimage of hosts:

['db2133.codfw.wmnet']

Of which those FAILED:

['db2133.codfw.wmnet']

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2133.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911140849_marostegui_198980.log.

Completed auto-reimage of hosts:

['db2133.codfw.wmnet']

and were ALL successful.

Change 550810 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Place db2133 into m2

https://gerrit.wikimedia.org/r/550810

Change 550810 merged by Marostegui:
[operations/puppet@production] mariadb: Place db2133 into m2

https://gerrit.wikimedia.org/r/550810

Mentioned in SAL (#wikimedia-operations) [2019-11-14T09:24:02Z] <marostegui> Stop mysql on db2067 to clone db21133 - T238183

Change 551036 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2134 into m3 codfw

https://gerrit.wikimedia.org/r/551036

Change 551036 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2134 into m3 codfw

https://gerrit.wikimedia.org/r/551036

Mentioned in SAL (#wikimedia-operations) [2019-11-15T06:41:23Z] <marostegui> Stop MySQL on db2065 to clone db2134 (this will trigger an haproxy irc alert) - T238183

Change 551371 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2135 into m5

https://gerrit.wikimedia.org/r/551371

Mentioned in SAL (#wikimedia-operations) [2019-11-18T07:09:19Z] <marostegui> Stop MySQL on db2070 to clone db2135 - T238183

Change 551371 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2135 into m5

https://gerrit.wikimedia.org/r/551371

Change 551959 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2132: Enable notifications

https://gerrit.wikimedia.org/r/551959

Change 551959 merged by Marostegui:
[operations/puppet@production] db2132: Enable notifications

https://gerrit.wikimedia.org/r/551959

Change 551961 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2132 to m1 codfw master

https://gerrit.wikimedia.org/r/551961

Change 551961 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2132 to m1 codfw master

https://gerrit.wikimedia.org/r/551961

Mentioned in SAL (#wikimedia-operations) [2019-11-20T07:43:02Z] <marostegui> Promote db2132 as m1-codfw master - T238183

db2132 is now m1 codfw master.
db2062 will be decommissioned

root@cumin1001:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hm1-master.codfw.wmnet -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db2132     |
+------------+
Marostegui updated the task description. (Show Details)Wed, Nov 20, 7:44 AM
Marostegui changed the status of subtask T233185: Decommission db2067.codfw.wmnet from Stalled to Open.Wed, Nov 20, 9:25 AM

Change 552173 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2133: Enable notifications

https://gerrit.wikimedia.org/r/552173

Change 552173 merged by Marostegui:
[operations/puppet@production] db2133: Enable notifications

https://gerrit.wikimedia.org/r/552173

Change 552191 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2133 to m2 codfw master

https://gerrit.wikimedia.org/r/552191

Change 552191 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2133 to m2 codfw master

https://gerrit.wikimedia.org/r/552191

Mentioned in SAL (#wikimedia-operations) [2019-11-21T07:56:22Z] <marostegui> Promote db2133 to codfw m2 master - T238183

db2133 is now m2 codfw master
db2067 will be decommissioned in a few days

mysql --skip-ssl -hm2-master.codfw.wmnet -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db2133     |
+------------+
Marostegui updated the task description. (Show Details)Thu, Nov 21, 8:18 AM

Change 552380 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage db213[2-5]

https://gerrit.wikimedia.org/r/552380

Change 552380 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage db213[2-5]

https://gerrit.wikimedia.org/r/552380

Change 552690 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2134: Enable notifications

https://gerrit.wikimedia.org/r/552690

Change 552690 merged by Marostegui:
[operations/puppet@production] db2134: Enable notifications

https://gerrit.wikimedia.org/r/552690

Change 552699 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2134 to m3 codfw master

https://gerrit.wikimedia.org/r/552699

Change 552699 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2134 to m3 codfw master

https://gerrit.wikimedia.org/r/552699

db2134 is now m3 codfw master
db2065 will be decommissioned in a few days

mysql --skip-ssl -hm3-master.codfw.wmnet -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db2134     |
+------------+
Marostegui updated the task description. (Show Details)Tue, Nov 26, 8:10 AM

Change 553240 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2135: Enable notifications

https://gerrit.wikimedia.org/r/553240

Change 553241 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db2135 to config

https://gerrit.wikimedia.org/r/553241

Change 553240 merged by Marostegui:
[operations/puppet@production] db2135: Enable notifications

https://gerrit.wikimedia.org/r/553240

Change 553241 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db2135 to config

https://gerrit.wikimedia.org/r/553241

Mentioned in SAL (#wikimedia-operations) [2019-11-27T06:01:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Add db2135 to the config T238183 (duration: 01m 11s)

Mentioned in SAL (#wikimedia-operations) [2019-11-27T06:02:12Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Add db2135 to the config T238183 (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2019-11-27T06:05:33Z] <marostegui> Promote db2135 to codfw m5 master T238183

m5 codfw's master is now db2135.

Marostegui closed this task as Resolved.Wed, Nov 27, 6:21 AM
Marostegui updated the task description. (Show Details)

All the hosts have been productionized and are now masters of mX on codfw.
Pending is decommissioning old hosts - that is being tracked at: T228258