Page MenuHomePhabricator

Productionize 22 new codfw database servers
Closed, ResolvedPublic

Description

More than a task, this is a tracking list to not "lose" not yet in-production servers and coordinate how to set them up.

These servers T162159 have to be used to:

  • Decom servers <db2030
  • Setup misc servers appropiately
  • Setup the eventual s8

State:

  • db2071: pooled as s1 api
  • db2072: pooled into s1
  • db2073: pooled in s4
  • db2074: pooled in s3
  • db2075: pooled in s5
  • db2076: pooled in s6
  • db2077: pooled in s7
  • db2078: pooled in m1 (probably too large for that, we may change it in the future or add more shards)
  • db2079: provisioned on s5 and later s8
  • db2080: provisioned on s5 and later s8
  • db2081: provisioned on s5 and later s8
  • db2082: provisioned on s5 and later s8
  • db2083: provisioned on s5 and later s8
  • db2084: provisioned on s5 and later s8 and used to test multi-instance (T178359)
  • db2085: provisioned on s1, s8 as recentchanges T178359
  • db2086: provisioned on s5, s7 as recentchanges T178359
  • db2087: provisioned on s6, s7 as recentchanges T178359
  • db2088: provisioned on s1, s2 as recentchanges T178359
  • db2089: provisioned on s6, s8 as recentchanges T178359
  • db2090: provisioned on s4 in exchange for 2 servers sent to misc
  • db2091: provisioned on s2, s4 as recentchanges T178359
  • db2092: provisioned on s1 as api T191275#4099560

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2017-10-17T07:53:40Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db2086 to the config - T170662 (duration: 00m 45s)

Marostegui updated the task description. (Show Details)Oct 17 2017, 10:08 AM
Marostegui updated the task description. (Show Details)Oct 19 2017, 6:07 AM

Change 385135 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add db2092 to s1 and s3 as rc

https://gerrit.wikimedia.org/r/385135

Change 385135 merged by Marostegui:
[operations/puppet@production] mariadb: Add db2092 to s1 and s3 as rc

https://gerrit.wikimedia.org/r/385135

Marostegui updated the task description. (Show Details)Oct 19 2017, 12:33 PM
Marostegui updated the task description. (Show Details)Oct 24 2017, 8:34 AM

Change 386135 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2088 with stretch

https://gerrit.wikimedia.org/r/386135

Change 386135 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2088 with stretch

https://gerrit.wikimedia.org/r/386135

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db2088.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201710240848_marostegui_19799_db2088_codfw_wmnet.log.

Completed auto-reimage of hosts:

['db2088.codfw.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Oct 25 2017, 10:35 AM

Change 386357 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2091 as stretch

https://gerrit.wikimedia.org/r/386357

Marostegui updated the task description. (Show Details)Oct 25 2017, 10:37 AM

Change 386357 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2091 as stretch

https://gerrit.wikimedia.org/r/386357

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db2091.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201710251040_marostegui_20773_db2091_codfw_wmnet.log.

Completed auto-reimage of hosts:

['db2091.codfw.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Oct 26 2017, 12:37 PM
Marostegui updated the task description. (Show Details)Oct 26 2017, 12:54 PM
Marostegui updated the task description. (Show Details)Oct 30 2017, 7:38 AM
Marostegui updated the task description. (Show Details)Oct 30 2017, 8:03 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Oct 30 2017, 11:17 AM

Change 387524 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2089 with stretch

https://gerrit.wikimedia.org/r/387524

Marostegui updated the task description. (Show Details)Oct 31 2017, 6:43 AM

Change 387524 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2089 with stretch

https://gerrit.wikimedia.org/r/387524

Marostegui updated the task description. (Show Details)Oct 31 2017, 11:19 AM
Marostegui updated the task description. (Show Details)Oct 31 2017, 2:12 PM
Marostegui updated the task description. (Show Details)Nov 3 2017, 2:01 PM

Change 392400 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Leave reimaginable only the db latest servers

https://gerrit.wikimedia.org/r/392400

Marostegui moved this task from In progress to Next on the DBA board.Nov 21 2017, 10:22 AM

Change 392400 merged by Jcrespo:
[operations/puppet@production] mariadb: Leave reimaginable only the db latest servers

https://gerrit.wikimedia.org/r/392400

Change 394573 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Undeploy db2092, use db1085 for s1 (remove s3 special slaves)

https://gerrit.wikimedia.org/r/394573

Change 394573 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Undeploy db2092, use db2085 for s1 (remove s3 special slaves)

https://gerrit.wikimedia.org/r/394573

Change 394588 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Set db2085 with s1 and s8; make db2092 pending to be deployed

https://gerrit.wikimedia.org/r/394588

jcrespo updated the task description. (Show Details)Dec 1 2017, 3:48 PM

Change 394588 merged by Jcrespo:
[operations/puppet@production] Set db2085 with s1 and s8; make db2092 pending to be deployed

https://gerrit.wikimedia.org/r/394588

Change 394591 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] Remove db2092; setup db1085 with s1 and s8

https://gerrit.wikimedia.org/r/394591

Change 394591 merged by Jcrespo:
[operations/software@master] Remove db2092; setup db1085 with s1 and s8

https://gerrit.wikimedia.org/r/394591

Change 394596 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Set db2092 as spare explicity

https://gerrit.wikimedia.org/r/394596

Change 394596 merged by Jcrespo:
[operations/puppet@production] mariadb: Set db2092 as spare explicity

https://gerrit.wikimedia.org/r/394596

Latest updates: T178359#3803600 , reflected on the description already.

Marostegui removed Marostegui as the assignee of this task.Jan 4 2018, 7:24 AM

Change 413328 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] install_server: Set db2090, db2073 to reimage as stretch

https://gerrit.wikimedia.org/r/413328

Change 413332 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db2073 for maintenance

https://gerrit.wikimedia.org/r/413332

Change 413328 merged by Jcrespo:
[operations/puppet@production] install_server: Set db2090, db2073 to reimage as stretch

https://gerrit.wikimedia.org/r/413328

Change 413332 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Depool db2073 for maintenance

https://gerrit.wikimedia.org/r/413332

Change 413335 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Add db2090 to s4 section

https://gerrit.wikimedia.org/r/413335

Change 413335 merged by Jcrespo:
[operations/puppet@production] mariadb: Add db2090 to s4 section

https://gerrit.wikimedia.org/r/413335

Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts:

['db2090.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201802221024_jynus_4608.log.

Change 413337 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: disable alerts on db2073, db2090 and assign them to s4

https://gerrit.wikimedia.org/r/413337

Change 413337 merged by Jcrespo:
[operations/puppet@production] mariadb: disable alerts on db2073, db2090 and assign them to s4

https://gerrit.wikimedia.org/r/413337

Completed auto-reimage of hosts:

['db2090.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts:

['db2073.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201802221220_jynus_30806.log.

Completed auto-reimage of hosts:

['db2073.codfw.wmnet']

and were ALL successful.

jcrespo updated the task description. (Show Details)Feb 22 2018, 7:08 PM

Change 413439 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool db2090 for the first time on s4

https://gerrit.wikimedia.org/r/413439

Change 413439 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Pool db2090 for the first time on s4

https://gerrit.wikimedia.org/r/413439

Change 417218 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s4.hosts: Add db2090

https://gerrit.wikimedia.org/r/417218

Change 417218 merged by jenkins-bot:
[operations/software@master] s4.hosts: Add db2090

https://gerrit.wikimedia.org/r/417218

Change 424994 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db2092 to s1

https://gerrit.wikimedia.org/r/424994

Change 424997 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2072

https://gerrit.wikimedia.org/r/424997

Change 424994 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2092 to s1

https://gerrit.wikimedia.org/r/424994

Change 424997 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2072

https://gerrit.wikimedia.org/r/424997

Mentioned in SAL (#wikimedia-operations) [2018-04-09T06:38:01Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2072 - T170662 (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2018-04-09T06:41:01Z] <marostegui> Stop MySQL on db2072 to clone db2092 from it - T170662

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2092.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804090650_marostegui_30453.log.

Change 425001 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reinstall db2092

https://gerrit.wikimedia.org/r/425001

Change 425001 merged by Marostegui:
[operations/puppet@production] install_server: Allow reinstall db2092

https://gerrit.wikimedia.org/r/425001

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2092.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804090727_marostegui_6390.log.

Completed auto-reimage of hosts:

['db2092.codfw.wmnet']

and were ALL successful.

Change 425013 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s1.hosts: Add db2092 to s1

https://gerrit.wikimedia.org/r/425013

Change 425013 merged by jenkins-bot:
[operations/software@master] s1.hosts: Add db2092 to s1

https://gerrit.wikimedia.org/r/425013

Change 425014 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db2092 to the config

https://gerrit.wikimedia.org/r/425014

Change 425014 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db2092 to the config

https://gerrit.wikimedia.org/r/425014

Mentioned in SAL (#wikimedia-operations) [2018-04-09T09:02:22Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db2092 to the config - T170662 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-04-09T09:03:38Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db2092 to the config - T170662 (duration: 00m 59s)

Change 425022 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2092.yaml: Enable notifications

https://gerrit.wikimedia.org/r/425022

Change 425022 merged by Marostegui:
[operations/puppet@production] db2092.yaml: Enable notifications

https://gerrit.wikimedia.org/r/425022

Change 425047 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Pool db2092 in s1

https://gerrit.wikimedia.org/r/425047

Change 425047 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Pool db2092 in s1

https://gerrit.wikimedia.org/r/425047

Mentioned in SAL (#wikimedia-operations) [2018-04-09T13:52:46Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Pool db2092 in s1 T170662 (duration: 00m 59s)

Marostegui closed this task as Resolved.Apr 9 2018, 1:54 PM
Marostegui updated the task description. (Show Details)

All the hosts are now pooled.