Page MenuHomePhabricator

Productionize db11[26-38]
Closed, ResolvedPublic

Description

The following hosts need to be productionized and will be used to 1:1 replace the ones that will be decommissioned at T217396: Decommission db1061-db1073

  • db1126 A1 Will replace db1071 (s8)
  • db1127 A3 Will replace db1069 (x1)
  • db1128 A5 Will replace db1072 (m3)
  • db1129 A8 Will replace db1066 (s2)
  • db1130 B3 Will replace db1070 (s5)
  • db1131 B5 Will replace db1061 (s6)
  • db1132 B8 Will replace db1065 (m2)
  • db1133 C3 Will replace db1073 (m5) T222731: Storage problems with new host db1133
  • db1134 C6 Will replace db1067 (s1)
  • db1135 C8 Will replace db1063 (m1)
  • db1136 D1 Will replace db1062 (s7)
  • db1137 D5 Will replace db1064 (x1)
  • db1138 D8 Will replace db1068 (s4)
  • Check they are on tendril and zarcillo (already checked for the following hosts: db1126,db1127,db1128,db1129,db1130,db1131,db1132,db1133,db1134,db1135,db1136,db1137,db1138)
  • Check tendril user (already checked for the following hosts: db1126,db1127,db1128,db1129,db1130,db1131,db1132,db1133,db1134,db1135,db1136,db1137,db1138)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 510639 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/510639

Change 510640 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1126 into s8

https://gerrit.wikimedia.org/r/510640

Change 510640 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1126 into s8

https://gerrit.wikimedia.org/r/510640

Change 510639 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/510639

Marostegui updated the task description. (Show Details)May 16 2019, 2:42 PM
Marostegui updated the task description. (Show Details)May 16 2019, 3:43 PM

Change 510792 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/510792

Change 510793 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Pool db1134 into s1

https://gerrit.wikimedia.org/r/510793

Change 510793 merged by Marostegui:
[operations/puppet@production] mariadb: Pool db1134 into s1

https://gerrit.wikimedia.org/r/510793

Change 510792 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/510792

Change 511365 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1126,db1134: Enable notifications

https://gerrit.wikimedia.org/r/511365

Change 511365 merged by Marostegui:
[operations/puppet@production] db1126,db1134: Enable notifications

https://gerrit.wikimedia.org/r/511365

Change 511371 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Provision db1126,db1134

https://gerrit.wikimedia.org/r/511371

Change 511371 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Provision db1126,db1134

https://gerrit.wikimedia.org/r/511371

Mentioned in SAL (#wikimedia-operations) [2019-05-20T05:38:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db1126 into s8, db1134 into s1 T222682 (duration: 00m 49s)

Mentioned in SAL (#wikimedia-operations) [2019-05-20T05:39:54Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1126 into s8, db1134 into s1 T222682 (duration: 00m 49s)

Marostegui updated the task description. (Show Details)May 20 2019, 5:46 AM
Marostegui updated the task description. (Show Details)May 20 2019, 5:49 AM
Marostegui updated the task description. (Show Details)

Change 511807 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1136 into s7

https://gerrit.wikimedia.org/r/511807

Change 511807 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1136 into s7

https://gerrit.wikimedia.org/r/511807

Change 512086 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1136: Enable notifications

https://gerrit.wikimedia.org/r/512086

Change 512086 merged by Marostegui:
[operations/puppet@production] db1136: Enable notifications

https://gerrit.wikimedia.org/r/512086

Change 512087 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db1136 into s7

https://gerrit.wikimedia.org/r/512087

Change 512087 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db1136 into s7

https://gerrit.wikimedia.org/r/512087

Mentioned in SAL (#wikimedia-operations) [2019-05-23T05:02:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1136 into s7 T222682 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2019-05-23T05:03:21Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db1136 into s7 T222682 (duration: 00m 55s)

Marostegui updated the task description. (Show Details)May 23 2019, 5:05 AM

Change 512099 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1128 into m3

https://gerrit.wikimedia.org/r/512099

Change 512099 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1128 into m3

https://gerrit.wikimedia.org/r/512099

Mentioned in SAL (#wikimedia-operations) [2019-05-23T07:11:46Z] <marostegui> Stop MySQL on db1117:3323 to clone db1128 T222682

Marostegui updated the task description. (Show Details)May 24 2019, 1:00 PM

Change 513945 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1128: Enable notifications

https://gerrit.wikimedia.org/r/513945

Change 513945 merged by Marostegui:
[operations/puppet@production] db1128: Enable notifications

https://gerrit.wikimedia.org/r/513945

Marostegui updated the task description. (Show Details)Jun 3 2019, 5:14 AM
Marostegui updated the task description. (Show Details)Jun 5 2019, 7:46 AM
Marostegui updated the task description. (Show Details)Jun 5 2019, 9:15 AM
Marostegui updated the task description. (Show Details)Jun 5 2019, 9:23 AM

Change 514663 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1132 into m2

https://gerrit.wikimedia.org/r/514663

Change 514663 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1132 into m2

https://gerrit.wikimedia.org/r/514663

Mentioned in SAL (#wikimedia-operations) [2019-06-06T08:38:32Z] <marostegui> Stop MySQL on db1117:3322 - this will trigger haproxy alerts - T222682

Change 515003 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db1132, enable notifications

https://gerrit.wikimedia.org/r/515003

Change 515003 merged by Marostegui:
[operations/puppet@production] mariadb: db1132, enable notifications

https://gerrit.wikimedia.org/r/515003

Marostegui updated the task description. (Show Details)Jun 7 2019, 5:12 AM

Change 517797 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1135

https://gerrit.wikimedia.org/r/517797

Change 517797 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1135

https://gerrit.wikimedia.org/r/517797

Mentioned in SAL (#wikimedia-operations) [2019-06-19T05:50:00Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1135 T222682 (duration: 00m 56s)

Change 518456 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1135

https://gerrit.wikimedia.org/r/518456

Change 518456 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1135

https://gerrit.wikimedia.org/r/518456

Mentioned in SAL (#wikimedia-operations) [2019-06-24T05:06:55Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1135 from config T222682 (duration: 01m 07s)

Mentioned in SAL (#wikimedia-operations) [2019-06-24T05:07:56Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db1135 from config T222682 (duration: 00m 55s)

Change 518457 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1135: Allow reimage

https://gerrit.wikimedia.org/r/518457

Change 518457 merged by Marostegui:
[operations/puppet@production] db1135: Allow reimage

https://gerrit.wikimedia.org/r/518457

Marostegui updated the task description. (Show Details)Mon, Jun 24, 5:23 AM
Marostegui updated the task description. (Show Details)

Change 518458 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1135 from s4 to m1

https://gerrit.wikimedia.org/r/518458

Change 518458 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1135 from s4 to m1

https://gerrit.wikimedia.org/r/518458

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1135.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906240541_marostegui_87345.log.

Completed auto-reimage of hosts:

['db1135.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-06-24T06:01:53Z] <marostegui> Stop MySQL on db1117:3321 to clone db1135 (haproxy alert will be triggered) - T222682

Marostegui updated the task description. (Show Details)Mon, Jun 24, 6:37 AM

Change 518645 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage db1135

https://gerrit.wikimedia.org/r/518645

Change 518645 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage db1135

https://gerrit.wikimedia.org/r/518645

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1133.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906250507_marostegui_110176.log.

Change 518875 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1135: Enable notifications

https://gerrit.wikimedia.org/r/518875

Change 518875 merged by Marostegui:
[operations/puppet@production] db1135: Enable notifications

https://gerrit.wikimedia.org/r/518875

Marostegui updated the task description. (Show Details)Tue, Jun 25, 5:13 AM

Completed auto-reimage of hosts:

['db1133.eqiad.wmnet']

and were ALL successful.

Change 518877 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1133 in m5

https://gerrit.wikimedia.org/r/518877

Change 518877 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db1133 in m5

https://gerrit.wikimedia.org/r/518877

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1133.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906250716_marostegui_137094.log.

Completed auto-reimage of hosts:

['db1133.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-06-25T12:49:41Z] <marostegui> Stop MySQL on db1117:m5 (checked dumps, they are done) to clone db1133 - T222682

Marostegui updated the task description. (Show Details)Tue, Jun 25, 2:26 PM

Change 519171 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Provision db1133 in m5

https://gerrit.wikimedia.org/r/519171

Change 519172 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1133: Enable notifications

https://gerrit.wikimedia.org/r/519172

Change 519172 merged by Marostegui:
[operations/puppet@production] db1133: Enable notifications

https://gerrit.wikimedia.org/r/519172

Change 519171 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Provision db1133 in m5

https://gerrit.wikimedia.org/r/519171

Mentioned in SAL (#wikimedia-operations) [2019-06-26T05:28:20Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Add db1133 into m5 depooled T222682 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2019-06-26T05:29:21Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Add db1133 into m5 depooled T222682 (duration: 00m 55s)

Marostegui closed this task as Resolved.Wed, Jun 26, 5:30 AM
Marostegui updated the task description. (Show Details)

All these hosts are now provisioned