Page MenuHomePhabricator
Create Task
Maniphest T222772

Productionize db2[103-120]
Closed, ResolvedPublic
Actions

Description

  • db2103 A3 s1
  • db2104 A5 s2
  • db2105 A6 s3
  • db2106 A8 s4
  • db2107 B1 s2
  • db2108 B3 s2
  • db2109 B5 s3
  • db2110 B6 s4
  • db2111 B6 s5
  • db2112 C1 s1
  • db2113 C3 s5
  • db2114 C5 s6 Fixed: T222753: db2114 hardware problem
  • db2115 C6 x1
  • db2116 C6 s1
  • db2117 D1 s6
  • db2118 D3 s7
  • db2119 D3 s4
  • db2120 D5 s7
  • Check production grants and events (already checked for the following hosts: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2114, db2115,db2116,db2117,db2118,db2119,db2120)
  • Check grants for dump user are not present (already checked for the following hosts: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2114, db2115,db2116,db2117,db2118,db2119,db2120)
  • Plan future masters and candidate masters to be done at T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment
  • Check they are on tendril and zarcillo (already checked for the following hosts: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2114, db2115,db2116,db2117,db2118,db2119,db2120)
  • Check tendril user (already checked for the following hosts: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2114, db2115,db2116,db2117,db2118,db2119,db2120)

Details

Related Gerrit Patches:
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Pool db2118,db2120 into s7
operations/puppet : productiondb2118,db2120: Enable notifications
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Pool db2114 into s6
operations/puppet : productionmariadb: Provision db2118,db2120 into s7
operations/puppet : productiondb2114: Enable notifications
operations/puppet : productiondb2114: Provision it into s6
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Pool db2111,db2113 into s5
operations/puppet : productiondb2111,db2113: Enable notifications
operations/puppet : productionmariadb: Provision db2111,db2113 into s5
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Pool db2106,db2110,db2119
operations/puppet : productiondb2106,db2110,db2119: Enable notifications
operations/puppet : productionmariadb: db2106,db2110,db2119 into s4
operations/mediawiki-config : masterdb-codfw,db-eqiad.php: Pool db2105,db2109 into s3
operations/puppet : productiondb2105,db2109: Enable notifications
operations/puppet : productioninstall_server: Change db2114's MAC
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Add new hosts
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Pool db2103,db2112,db2116
operations/puppet : productionmariadb: Provision db2105,db2109 into s3
operations/puppet : productionmariadb: Enable notifications
operations/puppet : productionmariadb: Provision several hosts on s2
operations/puppet : productiondb2103,db2112,db2116: Enable notifications
operations/puppet : productionmariadb: Productionize db2117 as an s6 replica
operations/puppet : productionmariadb: db2103,db2112,db2116 into s1
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Pool db2115 in x1
operations/puppet : productiondb2115: Enable notifications
operations/puppet : productionmariadb: Pool db2115 on x1

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.May 8 2019, 6:14 AM

Change 508755 merged by Marostegui:
[operations/puppet@production] mariadb: Pool db2115 on x1

https://gerrit.wikimedia.org/r/508755

gerritbot added a comment.May 8 2019, 7:22 AM

Change 508762 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2115: Enable notifications

https://gerrit.wikimedia.org/r/508762

gerritbot added a comment.May 8 2019, 7:22 AM

Change 508762 merged by Marostegui:
[operations/puppet@production] db2115: Enable notifications

https://gerrit.wikimedia.org/r/508762

gerritbot added a comment.May 8 2019, 7:28 AM

Change 508764 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2115 in x1

https://gerrit.wikimedia.org/r/508764

gerritbot added a comment.May 8 2019, 7:30 AM

Change 508764 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2115 in x1

https://gerrit.wikimedia.org/r/508764

Marostegui updated the task description. (Show Details)May 8 2019, 7:31 AM
Stashbot added a comment.May 8 2019, 7:32 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-08T07:32:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2115 into x1 T222772 (duration: 01m 09s)

Stashbot added a comment.May 8 2019, 7:33 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-08T07:33:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2115 into x1 T222772 (duration: 00m 56s)

gerritbot added a comment.May 8 2019, 7:56 AM

Change 508768 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db2103,db2112,db2116 into s1

https://gerrit.wikimedia.org/r/508768

gerritbot added a comment.May 8 2019, 8:00 AM

Change 508768 merged by Marostegui:
[operations/puppet@production] mariadb: db2103,db2112,db2116 into s1

https://gerrit.wikimedia.org/r/508768

gerritbot added a comment.May 8 2019, 3:18 PM

Change 508844 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2103,db2112,db2116: Enable notifications

https://gerrit.wikimedia.org/r/508844

gerritbot added a comment.May 8 2019, 3:47 PM

Change 508852 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Productionize db2117 as an s6 replica

https://gerrit.wikimedia.org/r/508852

gerritbot added a comment.May 8 2019, 3:55 PM

Change 508852 merged by Jcrespo:
[operations/puppet@production] mariadb: Productionize db2117 as an s6 replica

https://gerrit.wikimedia.org/r/508852

gerritbot added a comment.May 9 2019, 4:54 AM

Change 508844 merged by Marostegui:
[operations/puppet@production] db2103,db2112,db2116: Enable notifications

https://gerrit.wikimedia.org/r/508844

gerritbot added a comment.May 9 2019, 5:01 AM

Change 508974 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2103,db2112,db2116

https://gerrit.wikimedia.org/r/508974

gerritbot added a comment.May 9 2019, 5:05 AM

Change 508974 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2103,db2112,db2116

https://gerrit.wikimedia.org/r/508974

Stashbot added a comment.May 9 2019, 5:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-09T05:07:23Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2103, db2112 and db2116 into s1 T222772 (duration: 01m 22s)

Marostegui updated the task description. (Show Details)May 9 2019, 5:07 AM
Stashbot added a comment.May 9 2019, 5:09 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-09T05:09:12Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2103, db2112 and db2116 into s1 T222772 (duration: 01m 41s)

gerritbot added a comment.May 9 2019, 5:17 AM

Change 508975 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision several hosts on s2

https://gerrit.wikimedia.org/r/508975

gerritbot added a comment.May 9 2019, 5:20 AM

Change 508975 merged by Marostegui:
[operations/puppet@production] mariadb: Provision several hosts on s2

https://gerrit.wikimedia.org/r/508975

jcrespo updated the task description. (Show Details)May 9 2019, 6:56 AM
gerritbot added a comment.May 9 2019, 7:49 AM

Change 508992 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add new hosts

https://gerrit.wikimedia.org/r/508992

gerritbot added a comment.May 9 2019, 8:04 AM

Change 508992 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add new hosts

https://gerrit.wikimedia.org/r/508992

Stashbot added a comment.May 9 2019, 8:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-09T08:06:03Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db1129, db2104, db2107, db2108 T222772 T222682 (duration: 00m 59s)

Stashbot mentioned this in T222682: Productionize db11[26-38].May 9 2019, 8:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-09T08:07:14Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Provision db1129, db2104, db2107, db2108 T222772 T222682 (duration: 00m 57s)

Marostegui updated the task description. (Show Details)May 9 2019, 8:09 AM
gerritbot added a comment.May 9 2019, 8:09 AM

Change 508998 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Enable notifications

https://gerrit.wikimedia.org/r/508998

gerritbot added a comment.May 9 2019, 8:11 AM

Change 508998 merged by Marostegui:
[operations/puppet@production] mariadb: Enable notifications

https://gerrit.wikimedia.org/r/508998

gerritbot added a comment.May 9 2019, 8:25 AM

Change 509004 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2105,db2109 into s3

https://gerrit.wikimedia.org/r/509004

gerritbot added a comment.May 9 2019, 8:27 AM

Change 509004 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2105,db2109 into s3

https://gerrit.wikimedia.org/r/509004

Marostegui updated the task description. (Show Details)May 9 2019, 2:56 PM
ops-monitoring-bot added a comment.May 10 2019, 4:55 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2114.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201905100455_marostegui_89703.log.

gerritbot added a comment.May 10 2019, 5:15 AM

Change 509182 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Change db2114's MAC

https://gerrit.wikimedia.org/r/509182

gerritbot added a comment.May 10 2019, 5:16 AM

Change 509182 merged by Marostegui:
[operations/puppet@production] install_server: Change db2114's MAC

https://gerrit.wikimedia.org/r/509182

gerritbot added a comment.May 10 2019, 5:23 AM

Change 509184 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2105,db2109: Enable notifications

https://gerrit.wikimedia.org/r/509184

gerritbot added a comment.May 10 2019, 5:25 AM

Change 509184 merged by Marostegui:
[operations/puppet@production] db2105,db2109: Enable notifications

https://gerrit.wikimedia.org/r/509184

gerritbot added a comment.May 10 2019, 5:34 AM

Change 509185 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db.codfw,db-eqiad.php: Pool db2105,db2109 into s3

https://gerrit.wikimedia.org/r/509185

gerritbot added a comment.May 10 2019, 5:39 AM

Change 509185 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Pool db2105,db2109 into s3

https://gerrit.wikimedia.org/r/509185

ops-monitoring-bot added a comment.May 10 2019, 5:40 AM

Completed auto-reimage of hosts:

['db2114.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.May 10 2019, 5:40 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-10T05:40:26Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2105 db2109 into s3 T222772 (duration: 00m 52s)

Marostegui updated the task description. (Show Details)May 10 2019, 5:40 AM
Stashbot added a comment.May 10 2019, 5:41 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-10T05:41:23Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2105 db2109 into s3 T222772 (duration: 00m 49s)

Marostegui updated the task description. (Show Details)May 10 2019, 6:10 AM
Marostegui updated the task description. (Show Details)May 13 2019, 5:08 AM
gerritbot added a comment.May 13 2019, 5:13 AM

Change 509750 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db2106,db2110,db2119 into s4

https://gerrit.wikimedia.org/r/509750

gerritbot added a comment.May 13 2019, 5:20 AM

Change 509750 merged by Marostegui:
[operations/puppet@production] mariadb: db2106,db2110,db2119 into s4

https://gerrit.wikimedia.org/r/509750

Marostegui updated the task description. (Show Details)May 13 2019, 9:30 AM
gerritbot added a comment.May 14 2019, 5:00 AM

Change 510071 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2106,db2110,db2119: Enable notifications

https://gerrit.wikimedia.org/r/510071

gerritbot added a comment.May 14 2019, 5:02 AM

Change 510071 merged by Marostegui:
[operations/puppet@production] db2106,db2110,db2119: Enable notifications

https://gerrit.wikimedia.org/r/510071

gerritbot added a comment.May 16 2019, 5:11 AM

Change 510638 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2106,db2110,db2119

https://gerrit.wikimedia.org/r/510638

gerritbot added a comment.May 16 2019, 5:14 AM

Change 510638 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2106,db2110,db2119

https://gerrit.wikimedia.org/r/510638

Stashbot added a comment.May 16 2019, 5:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-16T05:17:01Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2106, db2110, db2119 into s4 - T222772 (duration: 00m 58s)

Stashbot added a comment.May 16 2019, 5:18 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-16T05:18:19Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2106, db2110, db2119 into s4 - T222772 (duration: 00m 56s)

Marostegui updated the task description. (Show Details)May 16 2019, 5:19 AM
gerritbot added a comment.May 16 2019, 7:48 AM

Change 510691 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2111,db2113 into s5

https://gerrit.wikimedia.org/r/510691

gerritbot added a comment.May 16 2019, 7:51 AM

Change 510691 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2111,db2113 into s5

https://gerrit.wikimedia.org/r/510691

Marostegui updated the task description. (Show Details)May 16 2019, 9:35 AM
Marostegui updated the task description. (Show Details)May 16 2019, 1:57 PM
Stashbot added a comment.May 16 2019, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-16T13:57:51Z] <marostegui> and recreate the following hosts in tendril: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2115,db2116,db2117,db2119 T222772

gerritbot added a comment.May 17 2019, 6:05 AM

Change 510799 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2111,db2113 into s5

https://gerrit.wikimedia.org/r/510799

gerritbot added a comment.May 17 2019, 6:16 AM

Change 510800 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2111,db2113: Enable notifications

https://gerrit.wikimedia.org/r/510800

gerritbot added a comment.May 17 2019, 6:17 AM

Change 510800 merged by Marostegui:
[operations/puppet@production] db2111,db2113: Enable notifications

https://gerrit.wikimedia.org/r/510800

gerritbot added a comment.May 17 2019, 6:34 AM

Change 510799 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2111,db2113 into s5

https://gerrit.wikimedia.org/r/510799

Stashbot added a comment.May 17 2019, 6:35 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-17T06:35:41Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2111 and db2113 into s5 T222772 (duration: 00m 50s)

Marostegui updated the task description. (Show Details)May 17 2019, 6:35 AM
Stashbot added a comment.May 17 2019, 6:36 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-17T06:36:37Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2111 and db2113 into s5 T222772 (duration: 00m 49s)

gerritbot added a comment.May 20 2019, 5:05 AM

Change 511367 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2114: Provision it into s6

https://gerrit.wikimedia.org/r/511367

gerritbot added a comment.May 20 2019, 5:08 AM

Change 511367 merged by Marostegui:
[operations/puppet@production] db2114: Provision it into s6

https://gerrit.wikimedia.org/r/511367

Marostegui updated the task description. (Show Details)May 20 2019, 7:14 AM
Marostegui updated the task description. (Show Details)May 20 2019, 7:34 AM
Marostegui updated the task description. (Show Details)May 20 2019, 8:01 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.May 21 2019, 4:52 AM

Change 511623 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2114: Enable notifications

https://gerrit.wikimedia.org/r/511623

gerritbot added a comment.May 21 2019, 4:53 AM

Change 511624 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2114 into s6

https://gerrit.wikimedia.org/r/511624

gerritbot added a comment.May 21 2019, 4:53 AM

Change 511623 merged by Marostegui:
[operations/puppet@production] db2114: Enable notifications

https://gerrit.wikimedia.org/r/511623

gerritbot added a comment.May 21 2019, 5:02 AM

Change 511625 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db2118,db2120 into s7

https://gerrit.wikimedia.org/r/511625

gerritbot added a comment.May 21 2019, 5:04 AM

Change 511625 merged by Marostegui:
[operations/puppet@production] mariadb: Provision db2118,db2120 into s7

https://gerrit.wikimedia.org/r/511625

gerritbot added a comment.May 21 2019, 5:11 AM

Change 511624 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2114 into s6

https://gerrit.wikimedia.org/r/511624

Stashbot added a comment.May 21 2019, 5:13 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-21T05:13:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2114 into s6 - T222772 (duration: 00m 51s)

Marostegui updated the task description. (Show Details)May 21 2019, 5:13 AM
Stashbot added a comment.May 21 2019, 5:14 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-21T05:14:23Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2114 into s6 - T222772 (duration: 00m 50s)

Marostegui updated the task description. (Show Details)May 21 2019, 7:33 AM
Marostegui updated the task description. (Show Details)May 21 2019, 10:03 AM
gerritbot added a comment.May 22 2019, 5:09 AM

Change 511805 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2118,db2120: Enable notifications

https://gerrit.wikimedia.org/r/511805

gerritbot added a comment.May 22 2019, 5:13 AM

Change 511806 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2118,db2120 into s7

https://gerrit.wikimedia.org/r/511806

gerritbot added a comment.May 22 2019, 5:13 AM

Change 511805 merged by Marostegui:
[operations/puppet@production] db2118,db2120: Enable notifications

https://gerrit.wikimedia.org/r/511805

gerritbot added a comment.May 22 2019, 5:23 AM

Change 511806 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db2118,db2120 into s7

https://gerrit.wikimedia.org/r/511806

Stashbot added a comment.May 22 2019, 5:25 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-22T05:25:05Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s)

Marostegui updated the task description. (Show Details)May 22 2019, 5:25 AM
Stashbot added a comment.May 22 2019, 5:26 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-22T05:26:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2118 and db2120 into s7 T222772 (duration: 00m 55s)

Marostegui closed this task as Resolved.May 22 2019, 5:26 AM

All these hosts have been populated and pooled.
Pending steps will be followed up at T220170: weights tackling, candidate masters conversions, moving old hosts around, decommission hosts...

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL