Page MenuHomePhabricator
Create Task
Maniphest T222682

Productionize db11[26-38]
Closed, ResolvedPublic
Actions

Description

The following hosts need to be productionized and will be used to 1:1 replace the ones that will be decommissioned at T217396: Decommission db1061-db1073

  • db1126 A1 Will replace db1071 (s8)
  • db1127 A3 Will replace db1069 (x1)
  • db1128 A5 Will replace db1072 (m3)
  • db1129 A8 Will replace db1066 (s2)
  • db1130 B3 Will replace db1070 (s5)
  • db1131 B5 Will replace db1061 (s6)
  • db1132 B8 Will replace db1065 (m2)
  • db1133 C3 Will replace db1073 (m5) T222731: Storage problems with new host db1133
  • db1134 C6 Will replace db1067 (s1)
  • db1135 C8 Will replace db1063 (m1)
  • db1136 D1 Will replace db1062 (s7)
  • db1137 D5 Will replace db1064 (x1)
  • db1138 D8 Will replace db1068 (s4)
  • Check they are on tendril and zarcillo (already checked for the following hosts: db1126,db1127,db1128,db1129,db1130,db1131,db1132,db1133,db1134,db1135,db1136,db1137,db1138)
  • Check tendril user (already checked for the following hosts: db1126,db1127,db1128,db1129,db1130,db1131,db1132,db1133,db1134,db1135,db1136,db1137,db1138)

Details

SubjectRepoBranchLines +/-
db-eqiad,db-codfw.php: Provision db1133 in m5operations/mediawiki-configmaster+3 -0
db1133: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Provision db1133 in m5operations/puppetproduction+8 -5
db1135: Enable notificationsoperations/puppetproduction+0 -1
install_server: Do not reimage db1135operations/puppetproduction+1 -1
mariadb: Move db1135 from s4 to m1operations/puppetproduction+12 -3
db1135: Allow reimageoperations/puppetproduction+2 -1
db-eqiad,db-codfw.php: Remove db1135operations/mediawiki-configmaster+0 -3
db-eqiad.php: Depool db1135operations/mediawiki-configmaster+1 -1
mariadb: db1132, enable notificationsoperations/puppetproduction+1 -1
mariadb: Provision db1132 into m2operations/puppetproduction+10 -0
db1128: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Provision db1128 into m3operations/puppetproduction+12 -1
db-eqiad,db-codfw.php: Pool db1136 into s7operations/mediawiki-configmaster+3 -0
db1136: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Provision db1136 into s7operations/puppetproduction+7 -2
db-codfw,db-eqiad.php: Provision db1126,db1134operations/mediawiki-configmaster+6 -0
db1126,db1134: Enable notificationsoperations/puppetproduction+0 -2
db-eqiad.php: Depool db1083operations/mediawiki-configmaster+1 -1
mariadb: Pool db1134 into s1operations/puppetproduction+7 -2
db-eqiad.php: Depool db1104operations/mediawiki-configmaster+5 -5
mariadb: Provision db1126 into s8operations/puppetproduction+7 -2
db-eqiad,db-codfw.php: Pool db1130, db1138operations/mediawiki-configmaster+6 -0
db1130,db1138: Enable notificationsoperations/puppetproduction+0 -2
mariadb: Provision db1130 into s5operations/puppetproduction+7 -2
db-eqiad.php: Pool db1129 in s2operations/mediawiki-configmaster+1 -0
db-eqiad,db-codfw.php: Add new hostsoperations/mediawiki-configmaster+14 -0
mariadb: Provision db1138 in s4operations/puppetproduction+7 -2
mariadb: Enable notificationsoperations/puppetproduction+0 -5
mariadb: Provision several hosts on s2operations/puppetproduction+21 -4
db-eqiad,db.codfw.php: Pool db1131 on s6operations/mediawiki-configmaster+3 -0
db1131: Enable notificationsoperations/puppetproduction+0 -1
db-eqiad,db-codfw.php: Pool db1127 and db1137 in x1operations/mediawiki-configmaster+7 -1
db1127,db1137: Enable notificationsoperations/puppetproduction+0 -2
mariadb: Provision db1131 on s6operations/puppetproduction+6 -1
mariadb: Provision db1127 and db1137 on x1operations/puppetproduction+20 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.May 16 2019, 5:25 AM

Change 510639 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/510639

gerritbot added a comment.May 16 2019, 5:25 AM

Change 510640 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1126 into s8

https://gerrit.wikimedia.org/r/510640

gerritbot added a comment.May 16 2019, 5:27 AM

Change 510640 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1126 into s8

https://gerrit.wikimedia.org/r/510640

gerritbot added a comment.May 16 2019, 5:27 AM

Change 510639 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/510639

• Marostegui updated the task description. (Show Details)May 16 2019, 2:42 PM
• Marostegui updated the task description. (Show Details)May 16 2019, 3:43 PM
gerritbot added a comment.May 17 2019, 5:12 AM

Change 510792 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/510792

gerritbot added a comment.May 17 2019, 5:13 AM

Change 510793 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Pool db1134 into s1

https://gerrit.wikimedia.org/r/510793

gerritbot added a comment.May 17 2019, 5:15 AM

Change 510793 merged by Marostegui:
[operations/puppet@production] mariadb: Pool db1134 into s1

https://gerrit.wikimedia.org/r/510793

gerritbot added a comment.May 17 2019, 5:16 AM

Change 510792 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/510792

gerritbot added a comment.May 20 2019, 4:57 AM

Change 511365 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1126,db1134: Enable notifications

https://gerrit.wikimedia.org/r/511365

gerritbot added a comment.May 20 2019, 4:59 AM

Change 511365 merged by Marostegui:
[operations/puppet@production] db1126,db1134: Enable notifications

https://gerrit.wikimedia.org/r/511365

gerritbot added a comment.May 20 2019, 5:27 AM

Change 511371 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Provision db1126,db1134

https://gerrit.wikimedia.org/r/511371

gerritbot added a comment.May 20 2019, 5:37 AM

Change 511371 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Provision db1126,db1134

https://gerrit.wikimedia.org/r/511371

Stashbot added a comment.May 20 2019, 5:38 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-20T05:38:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db1126 into s8, db1134 into s1 T222682 (duration: 00m 49s)

Stashbot added a comment.May 20 2019, 5:39 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-20T05:39:54Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1126 into s8, db1134 into s1 T222682 (duration: 00m 49s)

• Marostegui updated the task description. (Show Details)May 20 2019, 5:46 AM
• Marostegui updated the task description. (Show Details)May 20 2019, 5:49 AM
• Marostegui updated the task description. (Show Details)
gerritbot added a comment.May 22 2019, 5:33 AM

Change 511807 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1136 into s7

https://gerrit.wikimedia.org/r/511807

gerritbot added a comment.May 22 2019, 5:37 AM

Change 511807 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1136 into s7

https://gerrit.wikimedia.org/r/511807

Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:26 PM
gerritbot added a comment.May 23 2019, 4:45 AM

Change 512086 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1136: Enable notifications

https://gerrit.wikimedia.org/r/512086

gerritbot added a project: Patch-For-Review.May 23 2019, 4:45 AM
gerritbot added a comment.May 23 2019, 4:47 AM

Change 512086 merged by Marostegui:
[operations/puppet@production] db1136: Enable notifications

https://gerrit.wikimedia.org/r/512086

gerritbot added a comment.May 23 2019, 4:52 AM

Change 512087 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db1136 into s7

https://gerrit.wikimedia.org/r/512087

gerritbot added a comment.May 23 2019, 5:00 AM

Change 512087 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db1136 into s7

https://gerrit.wikimedia.org/r/512087

Stashbot added a comment.May 23 2019, 5:02 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-23T05:02:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1136 into s7 T222682 (duration: 00m 55s)

Stashbot added a comment.May 23 2019, 5:03 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-23T05:03:21Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db1136 into s7 T222682 (duration: 00m 55s)

• Marostegui updated the task description. (Show Details)May 23 2019, 5:05 AM
gerritbot added a comment.May 23 2019, 7:02 AM

Change 512099 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1128 into m3

https://gerrit.wikimedia.org/r/512099

gerritbot added a comment.May 23 2019, 7:09 AM

Change 512099 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1128 into m3

https://gerrit.wikimedia.org/r/512099

Stashbot added a comment.May 23 2019, 7:11 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-23T07:11:46Z] <marostegui> Stop MySQL on db1117:3323 to clone db1128 T222682

Maintenance_bot removed a project: Patch-For-Review.May 23 2019, 10:44 AM
• Marostegui updated the task description. (Show Details)May 24 2019, 1:00 PM
gerritbot added a comment.Jun 3 2019, 5:12 AM

Change 513945 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1128: Enable notifications

https://gerrit.wikimedia.org/r/513945

gerritbot added a project: Patch-For-Review.Jun 3 2019, 5:12 AM

Change 513945 merged by Marostegui:
[operations/puppet@production] db1128: Enable notifications

https://gerrit.wikimedia.org/r/513945

• Marostegui updated the task description. (Show Details)Jun 3 2019, 5:14 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 3 2019, 6:10 AM
• Marostegui mentioned this in T225060: db1091 crashed.Jun 5 2019, 7:24 AM
• Marostegui updated the task description. (Show Details)Jun 5 2019, 7:46 AM
• Marostegui updated the task description. (Show Details)Jun 5 2019, 9:15 AM
• Marostegui updated the task description. (Show Details)Jun 5 2019, 9:23 AM
gerritbot added a comment.Jun 6 2019, 8:08 AM

Change 514663 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1132 into m2

https://gerrit.wikimedia.org/r/514663

gerritbot added a project: Patch-For-Review.Jun 6 2019, 8:08 AM
gerritbot added a comment.Jun 6 2019, 8:33 AM

Change 514663 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1132 into m2

https://gerrit.wikimedia.org/r/514663

Stashbot added a comment.Jun 6 2019, 8:38 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-06T08:38:32Z] <marostegui> Stop MySQL on db1117:3322 - this will trigger haproxy alerts - T222682

gerritbot added a comment.Jun 7 2019, 5:09 AM

Change 515003 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db1132, enable notifications

https://gerrit.wikimedia.org/r/515003

gerritbot added a comment.Jun 7 2019, 5:10 AM

Change 515003 merged by Marostegui:
[operations/puppet@production] mariadb: db1132, enable notifications

https://gerrit.wikimedia.org/r/515003

• Marostegui updated the task description. (Show Details)Jun 7 2019, 5:12 AM
gerritbot added a comment.Jun 19 2019, 5:45 AM

Change 517797 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1135

https://gerrit.wikimedia.org/r/517797

gerritbot added a comment.Jun 19 2019, 5:48 AM

Change 517797 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1135

https://gerrit.wikimedia.org/r/517797

Stashbot added a comment.Jun 19 2019, 5:50 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-19T05:50:00Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1135 T222682 (duration: 00m 56s)

gerritbot added a comment.Jun 24 2019, 5:01 AM

Change 518456 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1135

https://gerrit.wikimedia.org/r/518456

gerritbot added a comment.Jun 24 2019, 5:04 AM

Change 518456 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1135

https://gerrit.wikimedia.org/r/518456

Stashbot added a comment.Jun 24 2019, 5:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-24T05:06:55Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1135 from config T222682 (duration: 01m 07s)

Stashbot added a comment.Jun 24 2019, 5:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-24T05:07:56Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db1135 from config T222682 (duration: 00m 55s)

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2019, 5:10 AM
gerritbot added a comment.Jun 24 2019, 5:13 AM

Change 518457 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1135: Allow reimage

https://gerrit.wikimedia.org/r/518457

gerritbot added a project: Patch-For-Review.Jun 24 2019, 5:13 AM

Change 518457 merged by Marostegui:
[operations/puppet@production] db1135: Allow reimage

https://gerrit.wikimedia.org/r/518457

• Marostegui updated the task description. (Show Details)Jun 24 2019, 5:23 AM
• Marostegui updated the task description. (Show Details)
gerritbot added a comment.Jun 24 2019, 5:27 AM

Change 518458 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1135 from s4 to m1

https://gerrit.wikimedia.org/r/518458

gerritbot added a comment.Jun 24 2019, 5:36 AM

Change 518458 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1135 from s4 to m1

https://gerrit.wikimedia.org/r/518458

ops-monitoring-bot added a comment.Jun 24 2019, 5:41 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1135.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906240541_marostegui_87345.log.

ops-monitoring-bot added a comment.Jun 24 2019, 5:59 AM

Completed auto-reimage of hosts:

['db1135.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 24 2019, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-24T06:01:53Z] <marostegui> Stop MySQL on db1117:3321 to clone db1135 (haproxy alert will be triggered) - T222682

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2019, 6:10 AM
• Marostegui updated the task description. (Show Details)Jun 24 2019, 6:37 AM
gerritbot added a comment.Jun 24 2019, 7:05 AM

Change 518645 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage db1135

https://gerrit.wikimedia.org/r/518645

gerritbot added a project: Patch-For-Review.Jun 24 2019, 7:05 AM

Change 518645 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage db1135

https://gerrit.wikimedia.org/r/518645

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2019, 7:10 AM
ops-monitoring-bot added a comment.Jun 25 2019, 5:07 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1133.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906250507_marostegui_110176.log.

gerritbot added a comment.Jun 25 2019, 5:10 AM

Change 518875 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1135: Enable notifications

https://gerrit.wikimedia.org/r/518875

gerritbot added a project: Patch-For-Review.Jun 25 2019, 5:10 AM

Change 518875 merged by Marostegui:
[operations/puppet@production] db1135: Enable notifications

https://gerrit.wikimedia.org/r/518875

• Marostegui updated the task description. (Show Details)Jun 25 2019, 5:13 AM
ops-monitoring-bot added a comment.Jun 25 2019, 5:28 AM

Completed auto-reimage of hosts:

['db1133.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jun 25 2019, 5:40 AM

Change 518877 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1133 in m5

https://gerrit.wikimedia.org/r/518877

gerritbot added a comment.Jun 25 2019, 7:15 AM

Change 518877 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1133 in m5

https://gerrit.wikimedia.org/r/518877

ops-monitoring-bot added a comment.Jun 25 2019, 7:16 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1133.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906250716_marostegui_137094.log.

ops-monitoring-bot added a comment.Jun 25 2019, 7:36 AM

Completed auto-reimage of hosts:

['db1133.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 25 2019, 12:49 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-25T12:49:41Z] <marostegui> Stop MySQL on db1117:m5 (checked dumps, they are done) to clone db1133 - T222682

Maintenance_bot removed a project: Patch-For-Review.Jun 25 2019, 1:10 PM
• Marostegui updated the task description. (Show Details)Jun 25 2019, 2:26 PM
gerritbot added a comment.Jun 26 2019, 4:52 AM

Change 519171 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Provision db1133 in m5

https://gerrit.wikimedia.org/r/519171

gerritbot added a project: Patch-For-Review.Jun 26 2019, 4:52 AM
gerritbot added a comment.Jun 26 2019, 4:54 AM

Change 519172 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1133: Enable notifications

https://gerrit.wikimedia.org/r/519172

gerritbot added a comment.Jun 26 2019, 4:55 AM

Change 519172 merged by Marostegui:
[operations/puppet@production] db1133: Enable notifications

https://gerrit.wikimedia.org/r/519172

gerritbot added a comment.Jun 26 2019, 5:26 AM

Change 519171 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Provision db1133 in m5

https://gerrit.wikimedia.org/r/519171

Stashbot added a comment.Jun 26 2019, 5:28 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-26T05:28:20Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Add db1133 into m5 depooled T222682 (duration: 00m 55s)

Stashbot added a comment.Jun 26 2019, 5:29 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-26T05:29:21Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Add db1133 into m5 depooled T222682 (duration: 00m 55s)

• Marostegui closed this task as Resolved.Jun 26 2019, 5:30 AM
• Marostegui updated the task description. (Show Details)

All these hosts are now provisioned

• Marostegui mentioned this in T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment.Jun 27 2019, 8:32 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL