Page MenuHomePhabricator

Productionize 2 new eqiad database servers
Closed, ResolvedPublic

Description

More than a task, this is a tracking list to not "lose" not yet in-production servers and coordinate how to set them up.

State:

  • db1113: pre-provisioned as db1113:3315 and db1113:3316 (not closing this task until they are fully provisioned in mediawiki)
  • db1114: provisioned in s1

Details

Related Gerrit Patches:
operations/mediawiki-config : masterdb-eqiad.php: Pool db1113:3316
operations/mediawiki-config : masterdb-eqiad.php: Pool db1113:3315
operations/mediawiki-config : masterdb-eqiad.php: Depool db1063
operations/mediawiki-config : masterdb-eqiad.php: Depool db1051
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Add initial config db1113
operations/software : masters5,s6.hosts: Add db1113
operations/puppet : productionmariadb: Initial setup for db1113
operations/puppet : productionsite.pp: Add db111{3,4} to spare

Event Timeline

Marostegui triaged this task as Medium priority.Jan 4 2018, 8:20 AM
Marostegui created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2018, 8:20 AM
Marostegui moved this task from Triage to Next on the DBA board.Jan 4 2018, 8:20 AM

Maybe solve T182896 if we leave this open?

Change 401905 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: Add db111{3,4} to spare

https://gerrit.wikimedia.org/r/401905

Maybe solve T182896 if we leave this open?

hehe yeah, I was about to :)

Change 401905 merged by Marostegui:
[operations/puppet@production] site.pp: Add db111{3,4} to spare

https://gerrit.wikimedia.org/r/401905

Marostegui removed Marostegui as the assignee of this task.Feb 9 2018, 12:18 PM
Marostegui updated the task description. (Show Details)Mar 9 2018, 10:29 AM

Change 417781 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision db1113

https://gerrit.wikimedia.org/r/417781

Change 417781 merged by Marostegui:
[operations/puppet@production] mariadb: Initial setup for db1113

https://gerrit.wikimedia.org/r/417781

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db1113.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201803091048_marostegui_30692_db1113_eqiad_wmnet.log.

Change 417796 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s5,s6.hosts: Add db1113

https://gerrit.wikimedia.org/r/417796

Completed auto-reimage of hosts:

['db1113.eqiad.wmnet']

and were ALL successful.

Change 417796 merged by jenkins-bot:
[operations/software@master] s5,s6.hosts: Add db1113

https://gerrit.wikimedia.org/r/417796

Change 417799 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/417799

Change 417799 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/417799

Mentioned in SAL (#wikimedia-operations) [2018-03-09T11:14:47Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1051 to clone db1113 - T184161 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-03-09T11:15:02Z] <marostegui> Stop MySQL on db1051 to clone db1113 - https://phabricator.wikimedia.org/T184161

Change 417812 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add initial config db1113

https://gerrit.wikimedia.org/r/417812

Change 417812 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add initial config db1113

https://gerrit.wikimedia.org/r/417812

Mentioned in SAL (#wikimedia-operations) [2018-03-09T11:34:48Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add initial config for db1113 multiinstance - T184161 (duration: 00m 58s)

Marostegui moved this task from Next to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2018-03-09T11:35:51Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add initial config for db1113 multiinstance - T184161 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-03-09T12:06:58Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db1051 after cloning db1113:3315- T184161 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-03-09T12:08:02Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1051 after cloning db1113:3315- T184161 (duration: 00m 58s)

Change 417832 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1063

https://gerrit.wikimedia.org/r/417832

Change 417832 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1063

https://gerrit.wikimedia.org/r/417832

Mentioned in SAL (#wikimedia-operations) [2018-03-09T12:16:39Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1063 to clone db1113:3316 - T184161 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-03-09T12:16:56Z] <marostegui> Stop mysql on db1063 to clone db1113:3316 - T184161

Mentioned in SAL (#wikimedia-operations) [2018-03-09T12:26:14Z] <marostegui> Compress s5 on db1113:3315 - T184161

Mentioned in SAL (#wikimedia-operations) [2018-03-09T13:12:19Z] <marostegui> Compress s6 on db1113:3316 - T184161

Marostegui updated the task description. (Show Details)Mar 9 2018, 1:17 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-09T15:20:23Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1063 after cloning db1113:3316 - T184161 (duration: 00m 58s)

Change 418847 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1113:3315

https://gerrit.wikimedia.org/r/418847

Change 418847 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1113:3315

https://gerrit.wikimedia.org/r/418847

Mentioned in SAL (#wikimedia-operations) [2018-03-12T06:54:37Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1113:3315 as vslow,dump in s5 - T184161 (duration: 00m 58s)

Change 418848 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1113:3316

https://gerrit.wikimedia.org/r/418848

Change 418848 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1113:3316

https://gerrit.wikimedia.org/r/418848

Mentioned in SAL (#wikimedia-operations) [2018-03-12T07:10:54Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1113:3316 as vslow,dump in s6 - T184161 (duration: 00m 58s)

Marostegui closed this task as Resolved.Mar 12 2018, 7:14 AM
Marostegui updated the task description. (Show Details)

db1113:3315 and db1113:3316 are now serving traffic.