Page MenuHomePhabricator
Create Task
Maniphest T238183

Productionize db213[2-5}
Closed, ResolvedPublic
Actions

Description

The following hosts are ready to go into misc services

  • db2132 A5 - m1
  • db2133 A6 - m2
  • db2134 B6 - m3
  • db2135 C6 - m5

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui triaged this task as Medium priority.Nov 13 2019, 6:55 AM
Marostegui created this task.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)Nov 13 2019, 7:15 AM
gerritbot added a comment.Nov 13 2019, 7:26 AM

Change 550589 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Pool db2132 into m1

https://gerrit.wikimedia.org/r/550589

gerritbot added a project: Patch-For-Review.Nov 13 2019, 7:26 AM
gerritbot added a comment.Nov 13 2019, 7:30 AM

Change 550589 merged by Marostegui:
[operations/puppet@production] mariadb: Pool db2132 into m1

https://gerrit.wikimedia.org/r/550589

gerritbot added a comment.Nov 13 2019, 7:56 AM

Change 550593 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2132 with stretch

https://gerrit.wikimedia.org/r/550593

gerritbot added a comment.Nov 13 2019, 7:59 AM

Change 550593 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2132 with stretch

https://gerrit.wikimedia.org/r/550593

ops-monitoring-bot added a comment.Nov 13 2019, 8:00 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2132.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911130800_marostegui_190803.log.

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2019, 8:10 AM
ops-monitoring-bot added a comment.Nov 13 2019, 8:25 AM

Completed auto-reimage of hosts:

['db2132.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Nov 13 2019, 8:25 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-13T08:25:32Z] <marostegui> Stop MySQL on db2062 to copy its data to db2132 T238183

Marostegui added a subtask: T233185: Decommission db2067.codfw.wmnet.Nov 13 2019, 10:03 AM
Marostegui claimed this task.Nov 13 2019, 3:53 PM
Marostegui added a subscriber: Papaul.
gerritbot added a comment.Nov 14 2019, 8:18 AM

Change 550801 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db213[3-5]

https://gerrit.wikimedia.org/r/550801

gerritbot added a project: Patch-For-Review.Nov 14 2019, 8:18 AM
gerritbot added a comment.Nov 14 2019, 8:20 AM

Change 550801 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db213[3-5]

https://gerrit.wikimedia.org/r/550801

ops-monitoring-bot added a comment.Nov 14 2019, 8:23 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2133.codfw.wmnet', 'db2134.codfw.wmnet', 'db2135.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911140823_marostegui_192468.log.

ops-monitoring-bot added a comment.Nov 14 2019, 8:46 AM

Completed auto-reimage of hosts:

['db2133.codfw.wmnet']

Of which those FAILED:

['db2133.codfw.wmnet']
ops-monitoring-bot added a comment.Nov 14 2019, 8:49 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2133.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911140849_marostegui_198980.log.

Maintenance_bot removed a project: Patch-For-Review.Nov 14 2019, 9:10 AM
ops-monitoring-bot added a comment.Nov 14 2019, 9:13 AM

Completed auto-reimage of hosts:

['db2133.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Nov 14 2019, 9:19 AM

Change 550810 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Place db2133 into m2

https://gerrit.wikimedia.org/r/550810

gerritbot added a project: Patch-For-Review.Nov 14 2019, 9:19 AM
gerritbot added a comment.Nov 14 2019, 9:22 AM

Change 550810 merged by Marostegui:
[operations/puppet@production] mariadb: Place db2133 into m2

https://gerrit.wikimedia.org/r/550810

Stashbot added a comment.Nov 14 2019, 9:24 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-14T09:24:02Z] <marostegui> Stop mysql on db2067 to clone db21133 - T238183

Maintenance_bot removed a project: Patch-For-Review.Nov 14 2019, 10:10 AM
gerritbot added a comment.Nov 15 2019, 6:37 AM

Change 551036 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2134 into m3 codfw

https://gerrit.wikimedia.org/r/551036

gerritbot added a project: Patch-For-Review.Nov 15 2019, 6:37 AM
gerritbot added a comment.Nov 15 2019, 6:40 AM

Change 551036 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2134 into m3 codfw

https://gerrit.wikimedia.org/r/551036

Stashbot added a comment.Nov 15 2019, 6:41 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-15T06:41:23Z] <marostegui> Stop MySQL on db2065 to clone db2134 (this will trigger an haproxy irc alert) - T238183

Maintenance_bot removed a project: Patch-For-Review.Nov 15 2019, 7:10 AM
gerritbot added a comment.Nov 18 2019, 7:06 AM

Change 551371 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2135 into m5

https://gerrit.wikimedia.org/r/551371

gerritbot added a project: Patch-For-Review.Nov 18 2019, 7:06 AM
Stashbot added a comment.Nov 18 2019, 7:09 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-18T07:09:19Z] <marostegui> Stop MySQL on db2070 to clone db2135 - T238183

gerritbot added a comment.Nov 18 2019, 7:10 AM

Change 551371 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2135 into m5

https://gerrit.wikimedia.org/r/551371

Maintenance_bot removed a project: Patch-For-Review.Nov 18 2019, 7:10 AM
gerritbot added a comment.Nov 20 2019, 7:28 AM

Change 551959 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2132: Enable notifications

https://gerrit.wikimedia.org/r/551959

gerritbot added a project: Patch-For-Review.Nov 20 2019, 7:28 AM
gerritbot added a comment.Nov 20 2019, 7:30 AM

Change 551959 merged by Marostegui:
[operations/puppet@production] db2132: Enable notifications

https://gerrit.wikimedia.org/r/551959

gerritbot added a comment.Nov 20 2019, 7:37 AM

Change 551961 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2132 to m1 codfw master

https://gerrit.wikimedia.org/r/551961

gerritbot added a comment.Nov 20 2019, 7:41 AM

Change 551961 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2132 to m1 codfw master

https://gerrit.wikimedia.org/r/551961

Stashbot added a comment.Nov 20 2019, 7:43 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-20T07:43:02Z] <marostegui> Promote db2132 as m1-codfw master - T238183

Marostegui added a comment.Nov 20 2019, 7:43 AM

db2132 is now m1 codfw master.
db2062 will be decommissioned

root@cumin1001:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hm1-master.codfw.wmnet -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db2132     |
+------------+
Marostegui updated the task description. (Show Details)Nov 20 2019, 7:44 AM
Maintenance_bot removed a project: Patch-For-Review.Nov 20 2019, 8:10 AM
Marostegui changed the status of subtask T233185: Decommission db2067.codfw.wmnet from Stalled to Open.Nov 20 2019, 9:25 AM
gerritbot added a comment.Nov 21 2019, 7:30 AM

Change 552173 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2133: Enable notifications

https://gerrit.wikimedia.org/r/552173

gerritbot added a project: Patch-For-Review.Nov 21 2019, 7:30 AM
gerritbot added a comment.Nov 21 2019, 7:32 AM

Change 552173 merged by Marostegui:
[operations/puppet@production] db2133: Enable notifications

https://gerrit.wikimedia.org/r/552173

gerritbot added a comment.Nov 21 2019, 7:47 AM

Change 552191 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2133 to m2 codfw master

https://gerrit.wikimedia.org/r/552191

gerritbot added a comment.Nov 21 2019, 7:53 AM

Change 552191 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2133 to m2 codfw master

https://gerrit.wikimedia.org/r/552191

Stashbot added a comment.Nov 21 2019, 7:56 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-21T07:56:22Z] <marostegui> Promote db2133 to codfw m2 master - T238183

Maintenance_bot removed a project: Patch-For-Review.Nov 21 2019, 8:10 AM
Marostegui added a comment.Nov 21 2019, 8:18 AM

db2133 is now m2 codfw master
db2067 will be decommissioned in a few days

mysql --skip-ssl -hm2-master.codfw.wmnet -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db2133     |
+------------+
Marostegui updated the task description. (Show Details)Nov 21 2019, 8:18 AM
gerritbot added a comment.Nov 22 2019, 6:15 AM

Change 552380 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage db213[2-5]

https://gerrit.wikimedia.org/r/552380

gerritbot added a project: Patch-For-Review.Nov 22 2019, 6:15 AM
gerritbot added a comment.Nov 22 2019, 6:18 AM

Change 552380 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage db213[2-5]

https://gerrit.wikimedia.org/r/552380

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2019, 7:10 AM
gerritbot added a comment.Nov 25 2019, 6:37 AM

Change 552690 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2134: Enable notifications

https://gerrit.wikimedia.org/r/552690

gerritbot added a project: Patch-For-Review.Nov 25 2019, 6:37 AM
gerritbot added a comment.Nov 25 2019, 6:42 AM

Change 552690 merged by Marostegui:
[operations/puppet@production] db2134: Enable notifications

https://gerrit.wikimedia.org/r/552690

gerritbot added a comment.Nov 25 2019, 7:05 AM

Change 552699 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2134 to m3 codfw master

https://gerrit.wikimedia.org/r/552699

gerritbot added a comment.Nov 25 2019, 7:13 AM

Change 552699 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2134 to m3 codfw master

https://gerrit.wikimedia.org/r/552699

Marostegui added a comment.Nov 25 2019, 7:26 AM

db2134 is now m3 codfw master
db2065 will be decommissioned in a few days

mysql --skip-ssl -hm3-master.codfw.wmnet -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db2134     |
+------------+
Marostegui updated the task description. (Show Details)Nov 26 2019, 8:10 AM
Maintenance_bot removed a project: Patch-For-Review.Nov 26 2019, 11:53 AM
gerritbot added a comment.Nov 27 2019, 5:54 AM

Change 553240 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2135: Enable notifications

https://gerrit.wikimedia.org/r/553240

gerritbot added a project: Patch-For-Review.Nov 27 2019, 5:54 AM
gerritbot added a comment.Nov 27 2019, 5:56 AM

Change 553241 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db2135 to config

https://gerrit.wikimedia.org/r/553241

gerritbot added a comment.Nov 27 2019, 5:56 AM

Change 553240 merged by Marostegui:
[operations/puppet@production] db2135: Enable notifications

https://gerrit.wikimedia.org/r/553240

gerritbot added a comment.Nov 27 2019, 5:59 AM

Change 553241 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db2135 to config

https://gerrit.wikimedia.org/r/553241

Stashbot added a comment.Nov 27 2019, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-27T06:01:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Add db2135 to the config T238183 (duration: 01m 11s)

Stashbot added a comment.Nov 27 2019, 6:02 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-27T06:02:12Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Add db2135 to the config T238183 (duration: 00m 59s)

Stashbot added a comment.Nov 27 2019, 6:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-27T06:05:33Z] <marostegui> Promote db2135 to codfw m5 master T238183

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2019, 6:10 AM
Marostegui added a comment.Nov 27 2019, 6:21 AM

m5 codfw's master is now db2135.

Marostegui closed this task as Resolved.Nov 27 2019, 6:21 AM
Marostegui updated the task description. (Show Details)

All the hosts have been productionized and are now masters of mX on codfw.
Pending is decommissioning old hosts - that is being tracked at: T228258

Papaul closed subtask T233185: Decommission db2067.codfw.wmnet as Resolved.Dec 6 2019, 12:22 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits