Page MenuHomePhabricator

Productionize db2[103-120]
Closed, ResolvedPublic

Description

  • db2103 A3 s1
  • db2104 A5 s2
  • db2105 A6 s3
  • db2106 A8 s4
  • db2107 B1 s2
  • db2108 B3 s2
  • db2109 B5 s3
  • db2110 B6 s4
  • db2111 B6 s5
  • db2112 C1 s1
  • db2113 C3 s5
  • db2114 C5 s6 Fixed: T222753: db2114 hardware problem
  • db2115 C6 x1
  • db2116 C6 s1
  • db2117 D1 s6
  • db2118 D3 s7
  • db2119 D3 s4
  • db2120 D5 s7
  • Check production grants and events (already checked for the following hosts: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2114, db2115,db2116,db2117,db2118,db2119,db2120)
  • Check grants for dump user are not present (already checked for the following hosts: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2114, db2115,db2116,db2117,db2118,db2119,db2120)
  • Plan future masters and candidate masters to be done at T220170: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment
  • Check they are on tendril and zarcillo (already checked for the following hosts: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2114, db2115,db2116,db2117,db2118,db2119,db2120)
  • Check tendril user (already checked for the following hosts: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2114, db2115,db2116,db2117,db2118,db2119,db2120)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 508755 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Pool db2115 on x1

https://gerrit.wikimedia.org/r/508755

Change 508755 merged by Marostegui:
[operations/puppet@production] mariadb: Pool db2115 on x1

https://gerrit.wikimedia.org/r/508755

Change 508762 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2115: Enable notifications

https://gerrit.wikimedia.org/r/508762

Change 508762 merged by Marostegui:
[operations/puppet@production] db2115: Enable notifications

https://gerrit.wikimedia.org/r/508762

Change 508764 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2115 in x1

https://gerrit.wikimedia.org/r/508764

Change 508764 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2115 in x1

https://gerrit.wikimedia.org/r/508764

Marostegui updated the task description. (Show Details)May 8 2019, 7:31 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-08T07:32:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2115 into x1 T222772 (duration: 01m 09s)

Mentioned in SAL (#wikimedia-operations) [2019-05-08T07:33:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2115 into x1 T222772 (duration: 00m 56s)

Change 508768 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db2103,db2112,db2116 into s1

https://gerrit.wikimedia.org/r/508768

Change 508768 merged by Marostegui:
[operations/puppet@production] mariadb: db2103,db2112,db2116 into s1

https://gerrit.wikimedia.org/r/508768

Change 508844 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2103,db2112,db2116: Enable notifications

https://gerrit.wikimedia.org/r/508844

Change 508852 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Productionize db2117 as an s6 replica

https://gerrit.wikimedia.org/r/508852

Change 508852 merged by Jcrespo:
[operations/puppet@production] mariadb: Productionize db2117 as an s6 replica

https://gerrit.wikimedia.org/r/508852

Change 508844 merged by Marostegui:
[operations/puppet@production] db2103,db2112,db2116: Enable notifications

https://gerrit.wikimedia.org/r/508844

Change 508974 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2103,db2112,db2116

https://gerrit.wikimedia.org/r/508974

Change 508974 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2103,db2112,db2116

https://gerrit.wikimedia.org/r/508974

Mentioned in SAL (#wikimedia-operations) [2019-05-09T05:07:23Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2103, db2112 and db2116 into s1 T222772 (duration: 01m 22s)

Marostegui updated the task description. (Show Details)May 9 2019, 5:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-09T05:09:12Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2103, db2112 and db2116 into s1 T222772 (duration: 01m 41s)

Change 508975 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision several hosts on s2

https://gerrit.wikimedia.org/r/508975

Change 508975 merged by Marostegui:
[operations/puppet@production] mariadb: Provision several hosts on s2

https://gerrit.wikimedia.org/r/508975

jcrespo updated the task description. (Show Details)May 9 2019, 6:56 AM

Change 508992 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add new hosts

https://gerrit.wikimedia.org/r/508992

Change 508992 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add new hosts

https://gerrit.wikimedia.org/r/508992

Mentioned in SAL (#wikimedia-operations) [2019-05-09T08:06:03Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db1129, db2104, db2107, db2108 T222772 T222682 (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2019-05-09T08:07:14Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Provision db1129, db2104, db2107, db2108 T222772 T222682 (duration: 00m 57s)

Marostegui updated the task description. (Show Details)May 9 2019, 8:09 AM

Change 508998 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Enable notifications

https://gerrit.wikimedia.org/r/508998

Change 508998 merged by Marostegui:
[operations/puppet@production] mariadb: Enable notifications

https://gerrit.wikimedia.org/r/508998

Change 509004 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2105,db2109 into s3

https://gerrit.wikimedia.org/r/509004

Change 509004 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2105,db2109 into s3

https://gerrit.wikimedia.org/r/509004

Marostegui updated the task description. (Show Details)May 9 2019, 2:56 PM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2114.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201905100455_marostegui_89703.log.

Change 509182 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Change db2114's MAC

https://gerrit.wikimedia.org/r/509182

Change 509182 merged by Marostegui:
[operations/puppet@production] install_server: Change db2114's MAC

https://gerrit.wikimedia.org/r/509182

Change 509184 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2105,db2109: Enable notifications

https://gerrit.wikimedia.org/r/509184

Change 509184 merged by Marostegui:
[operations/puppet@production] db2105,db2109: Enable notifications

https://gerrit.wikimedia.org/r/509184

Change 509185 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db.codfw,db-eqiad.php: Pool db2105,db2109 into s3

https://gerrit.wikimedia.org/r/509185

Change 509185 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Pool db2105,db2109 into s3

https://gerrit.wikimedia.org/r/509185

Completed auto-reimage of hosts:

['db2114.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-05-10T05:40:26Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2105 db2109 into s3 T222772 (duration: 00m 52s)

Marostegui updated the task description. (Show Details)May 10 2019, 5:40 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-10T05:41:23Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2105 db2109 into s3 T222772 (duration: 00m 49s)

Marostegui updated the task description. (Show Details)May 10 2019, 6:10 AM
Marostegui updated the task description. (Show Details)May 13 2019, 5:08 AM

Change 509750 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db2106,db2110,db2119 into s4

https://gerrit.wikimedia.org/r/509750

Change 509750 merged by Marostegui:
[operations/puppet@production] mariadb: db2106,db2110,db2119 into s4

https://gerrit.wikimedia.org/r/509750

Marostegui updated the task description. (Show Details)May 13 2019, 9:30 AM

Change 510071 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2106,db2110,db2119: Enable notifications

https://gerrit.wikimedia.org/r/510071

Change 510071 merged by Marostegui:
[operations/puppet@production] db2106,db2110,db2119: Enable notifications

https://gerrit.wikimedia.org/r/510071

Change 510638 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2106,db2110,db2119

https://gerrit.wikimedia.org/r/510638

Change 510638 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2106,db2110,db2119

https://gerrit.wikimedia.org/r/510638

Mentioned in SAL (#wikimedia-operations) [2019-05-16T05:17:01Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2106, db2110, db2119 into s4 - T222772 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2019-05-16T05:18:19Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2106, db2110, db2119 into s4 - T222772 (duration: 00m 56s)

Marostegui updated the task description. (Show Details)May 16 2019, 5:19 AM

Change 510691 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2111,db2113 into s5

https://gerrit.wikimedia.org/r/510691

Change 510691 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2111,db2113 into s5

https://gerrit.wikimedia.org/r/510691

Marostegui updated the task description. (Show Details)May 16 2019, 9:35 AM
Marostegui updated the task description. (Show Details)May 16 2019, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-16T13:57:51Z] <marostegui> and recreate the following hosts in tendril: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2115,db2116,db2117,db2119 T222772

Change 510799 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2111,db2113 into s5

https://gerrit.wikimedia.org/r/510799

Change 510800 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2111,db2113: Enable notifications

https://gerrit.wikimedia.org/r/510800

Change 510800 merged by Marostegui:
[operations/puppet@production] db2111,db2113: Enable notifications

https://gerrit.wikimedia.org/r/510800

Change 510799 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2111,db2113 into s5

https://gerrit.wikimedia.org/r/510799

Mentioned in SAL (#wikimedia-operations) [2019-05-17T06:35:41Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2111 and db2113 into s5 T222772 (duration: 00m 50s)

Marostegui updated the task description. (Show Details)May 17 2019, 6:35 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-17T06:36:37Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2111 and db2113 into s5 T222772 (duration: 00m 49s)

Change 511367 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2114: Provision it into s6

https://gerrit.wikimedia.org/r/511367

Change 511367 merged by Marostegui:
[operations/puppet@production] db2114: Provision it into s6

https://gerrit.wikimedia.org/r/511367

Marostegui updated the task description. (Show Details)May 20 2019, 7:14 AM
Marostegui updated the task description. (Show Details)May 20 2019, 7:34 AM
Marostegui updated the task description. (Show Details)May 20 2019, 8:01 AM
Marostegui updated the task description. (Show Details)

Change 511623 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2114: Enable notifications

https://gerrit.wikimedia.org/r/511623

Change 511624 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2114 into s6

https://gerrit.wikimedia.org/r/511624

Change 511623 merged by Marostegui:
[operations/puppet@production] db2114: Enable notifications

https://gerrit.wikimedia.org/r/511623

Change 511625 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2118,db2120 into s7

https://gerrit.wikimedia.org/r/511625

Change 511625 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2118,db2120 into s7

https://gerrit.wikimedia.org/r/511625

Change 511624 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2114 into s6

https://gerrit.wikimedia.org/r/511624

Mentioned in SAL (#wikimedia-operations) [2019-05-21T05:13:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2114 into s6 - T222772 (duration: 00m 51s)

Marostegui updated the task description. (Show Details)May 21 2019, 5:13 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-21T05:14:23Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2114 into s6 - T222772 (duration: 00m 50s)

Marostegui updated the task description. (Show Details)May 21 2019, 7:33 AM
Marostegui updated the task description. (Show Details)May 21 2019, 10:03 AM

Change 511805 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2118,db2120: Enable notifications

https://gerrit.wikimedia.org/r/511805

Change 511806 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2118,db2120 into s7

https://gerrit.wikimedia.org/r/511806

Change 511805 merged by Marostegui:
[operations/puppet@production] db2118,db2120: Enable notifications

https://gerrit.wikimedia.org/r/511805

Change 511806 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2118,db2120 into s7

https://gerrit.wikimedia.org/r/511806

Mentioned in SAL (#wikimedia-operations) [2019-05-22T05:25:05Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s)

Marostegui updated the task description. (Show Details)May 22 2019, 5:25 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-22T05:26:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s)

Marostegui closed this task as Resolved.May 22 2019, 5:26 AM

All these hosts have been populated and pooled.
Pending steps will be followed up at T220170: weights tackling, candidate masters conversions, moving old hosts around, decommission hosts...