Page MenuHomePhabricator
Create Task
Maniphest T172679

Productionize 11 new eqiad database servers
Closed, ResolvedPublic
Actions

Description

Like T170662, more than a task, this is a tracking list to not "lose" not yet in-production servers and coordinate how to set them up.

These servers T162233 have to be used to:

  • Decom servers < db1051
  • Setup misc servers appropiately
  • Setup the eventual s8

State:

  • db1096: provisioned on s5 rc service (and will later serve s8)
  • db1097: provisioned and serving s4
  • db1098: provisioned and serving s6
  • db1099: provisioned on s5 rc service (and will later serve s8)
  • db1100: provisioned on s5 (cloned from old master db1049 - and will later serve s8)
  • db1101: provisioned on s2
  • db1102: temporarily used as sanitarium3 - T169510
  • db1103: temporarily used in s3 to replace db1035 (cloned from db1035 to preserve its data)
  • db1104: provisioned on s5 (and will later serve s8)
  • db1105: provisioned on s5 (and will later serve s8)
  • db1106: provisioned on s5 (and will later serve s8)

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+1 -1db-eqiad.php: Depool db1103
operations/softwaremaster+1 -1s3.hosts,s1.hosts: Move db1072 from s1 to s3
operations/mediawiki-configmaster+4 -4db-eqiad.php: Depool db1038
operations/puppetproduction+4 -5mariadb: Move db1072 to s3
operations/mediawiki-configmaster+3 -3db-eqiad.php: Restore db1066 on s1, db1072 to s3
operations/puppetproduction+4 -5mariadb: Move db1066 from s1 to s3
operations/mediawiki-configmaster+1 -1db-eqiad.php: Move db1066 from s1 api to s3 vslow
operations/mediawiki-configmaster+4 -1db-eqiad,db-codfw.php: Add db1106 to config
operations/puppetproduction+3 -1mariadb: Add db1106 to s5
operations/softwaremaster+1 -0s5.hosts: Add db1106 to s5
operations/mediawiki-configmaster+4 -1db-eqiad,db-codfw.php: Add db1105 to the config
operations/softwaremaster+1 -0s5.hosts: Add db1105
operations/puppetproduction+3 -1mariadb: Add db1105 to s5
operations/mediawiki-configmaster+3 -0db-eqiad,db-codfw.php: Add db1104 to the config
operations/softwaremaster+1 -0s5.hosts: Add db1104 to s5
operations/puppetproduction+3 -1mariadb: Add db1104 to s5 cloned from db1087
operations/mediawiki-configmaster+1 -1db-eqiad.php: Depool db1087 to clone db1104
operations/mediawiki-configmaster+3 -3db-eqiad.php: Depool db1066
operations/mediawiki-configmaster+2 -0db-eqiad,db-codfw.php: Add db1103 to the array
operations/softwaremaster+1 -0s3.hosts: Add db1103
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1035 to clone it to db1103
operations/puppetproduction+3 -1mariadb: Add db1103 to s3
operations/puppetproduction+1 -1icinga: Disable notifications on db2078, enable them on db1101
operations/softwaremaster+1 -0dbtools: Add db1101 to s2
operations/mediawiki-configmaster+1 -1mariadb: Pool db1101 as new s2 host
operations/puppetproduction+4 -1mariadb: Setup db1101 on s2 to replace db1018 and db1036
operations/mediawiki-configmaster+1 -1db-eqiad.php: Pool db1100 with weight 0
operations/mediawiki-configmaster+1 -0db-eqiad.php: Add db1100 to s5 depooled
operations/mediawiki-configmaster+2 -0db-eqiad,db-codfw.php: Add db1100
operations/puppetproduction+3 -1mariadb: Add db1100 to s5
operations/softwaremaster+1 -0s5.hosts: Add db1100
operations/mediawiki-configmaster+2 -0db-eqiad.php: Fully pool db1099
operations/mediawiki-configmaster+2 -0db-eqiad.php: Add db1099 to s5
operations/softwaremaster+1 -0s5.hosts: Add db1099 to s5
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1045
operations/puppetproduction+3 -1mariadb: Add db1099 to s5
operations/mediawiki-configmaster+1 -1db-eqiad.php: Add weight to db1096
operations/mediawiki-configmaster+2 -2db-eqiad.php: Give some weight to db1096
operations/mediawiki-configmaster+6 -0db-eqiad.php: Add db1096 to s5
operations/softwaremaster+1 -0s5.hosts: Add db1096
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1026
operations/puppetproduction+3 -3mariadb: Add db1096 as s5 slave
operations/mediawiki-configmaster+8 -0mariadb: Add db1098 as new s6 recentchanges/watchlist/... replica
operations/softwaremaster+2 -1mariadb: Add db1098 to the list of available s6 hosts
operations/puppetproduction+3 -1mariadb: Pool db1098 as new s6 recentchanges/watchlist host
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Sep 7 2017, 7:54 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-07T07:54:51Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db1100 - T172679 (duration: 00m 48s)

gerritbot added a comment.Sep 7 2017, 9:09 AM

Change 376484 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Add db1100 to s5

https://gerrit.wikimedia.org/r/376484

gerritbot added a comment.Sep 7 2017, 9:14 AM

Change 376484 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Add db1100 to s5 depooled

https://gerrit.wikimedia.org/r/376484

Stashbot added a comment.Sep 7 2017, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-07T09:15:57Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db1100 depooled to s5 array - T172679 (duration: 00m 49s)

Marostegui updated the task description. (Show Details)Sep 7 2017, 9:16 AM
gerritbot added a comment.Sep 7 2017, 11:20 AM

Change 376499 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1100 with weight 0

https://gerrit.wikimedia.org/r/376499

gerritbot added a comment.Sep 7 2017, 11:23 AM

Change 376499 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1100 with weight 0

https://gerrit.wikimedia.org/r/376499

Stashbot added a comment.Sep 7 2017, 11:24 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-07T11:24:38Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1100 with 0 weight - T172679 (duration: 00m 49s)

gerritbot added a comment.Sep 19 2017, 9:50 AM

Change 378859 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Setup db1101 on s2 to replace db1018 and db1036

https://gerrit.wikimedia.org/r/378859

jcrespo updated the task description. (Show Details)Sep 19 2017, 9:51 AM
gerritbot added a comment.Sep 19 2017, 10:04 AM

Change 378859 merged by Jcrespo:
[operations/puppet@production] mariadb: Setup db1101 on s2 to replace db1018 and db1036

https://gerrit.wikimedia.org/r/378859

gerritbot added a comment.Sep 19 2017, 2:04 PM

Change 378914 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool db1101 as new s2 host

https://gerrit.wikimedia.org/r/378914

gerritbot added a comment.Sep 19 2017, 2:09 PM

Change 378916 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dbtools: Add db1101 to s2

https://gerrit.wikimedia.org/r/378916

gerritbot added a comment.Sep 19 2017, 4:06 PM

Change 378916 merged by Jcrespo:
[operations/software@master] dbtools: Add db1101 to s2

https://gerrit.wikimedia.org/r/378916

gerritbot added a comment.Sep 19 2017, 4:22 PM

Change 378914 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Pool db1101 as new s2 host

https://gerrit.wikimedia.org/r/378914

gerritbot added a comment.Sep 19 2017, 4:54 PM

Change 378962 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] icinga: Disable notifications on db2078, enable them on db1101

https://gerrit.wikimedia.org/r/378962

gerritbot added a comment.Sep 19 2017, 5:10 PM

Change 378962 merged by Jcrespo:
[operations/puppet@production] icinga: Disable notifications on db2078, enable them on db1101

https://gerrit.wikimedia.org/r/378962

Marostegui added a comment.Sep 26 2017, 8:12 AM

I would like to decommission db1035 from s3, it is low on disk space.
According to https://gerrit.wikimedia.org/r/#/c/338996/1/wmf-config/db-eqiad.php what we could do is..simply decommission it and get both db1077 and db1078 to serve main traffic + recentchanges service.
If not, we can get one of the non used hosts of this tasks, temporarily, clone it from db1035 and just decommission it and later decide what to do with the rc service for s3

@jcrespo any thoughts on this?

jcrespo added a comment.Sep 26 2017, 2:07 PM

Yes, although we may need still an extra host for vlow/dumps, separate from the other services, and smaller in size. Any ideas about which one we can move that is relatively old but not about to be removed?

Marostegui added a comment.Sep 27 2017, 8:08 AM
In T172679#3635468, @jcrespo wrote:

Yes, although we may need still an extra host for vlow/dumps, separate from the other services, and smaller in size. Any ideas about which one we can move that is relatively old but not about to be removed?

According to: https://gerrit.wikimedia.org/r/#/c/338996/1/wmf-config/db-eqiad.php

In s1 we'd have 2 of the most powerful servers serving API.
Currently we have 3x160G serving API, we could take one of those and get one of the big ones to serve API instead.

Marostegui added a comment.Sep 27 2017, 9:21 AM
In T172679#3634595, @Marostegui wrote:

I would like to decommission db1035 from s3, it is low on disk space.
According to https://gerrit.wikimedia.org/r/#/c/338996/1/wmf-config/db-eqiad.php what we could do is..simply decommission it and get both db1077 and db1078 to serve main traffic + recentchanges service.
If not, we can get one of the non used hosts of this tasks, temporarily, clone it from db1035 and just decommission it and later decide what to do with the rc service for s3

@jcrespo any thoughts on this?

Going to grab one of the servers and clone it from db1035, so we can save db1035's and later checksum it with no rush as db1035 is quickly filling up:

Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs   1.6T  1.5T  171G  90% /srv
Marostegui updated the task description. (Show Details)Sep 27 2017, 9:22 AM
gerritbot added a comment.Sep 27 2017, 9:26 AM

Change 380937 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add db1103 to s3

https://gerrit.wikimedia.org/r/380937

gerritbot added a comment.Sep 27 2017, 9:30 AM

Change 380938 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1035 to clone it to db1103

https://gerrit.wikimedia.org/r/380938

gerritbot added a comment.Sep 27 2017, 9:32 AM

Change 380937 merged by Marostegui:
[operations/puppet@production] mariadb: Add db1103 to s3

https://gerrit.wikimedia.org/r/380937

gerritbot added a comment.Sep 27 2017, 9:41 AM

Change 380938 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1035 to clone it to db1103

https://gerrit.wikimedia.org/r/380938

gerritbot added a comment.Sep 27 2017, 9:42 AM

Change 380944 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s3.hosts: Add db1103

https://gerrit.wikimedia.org/r/380944

Stashbot added a comment.Sep 27 2017, 9:42 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-27T09:42:37Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1035 to transfer its data to db1103 - T172679 (duration: 00m 48s)

Stashbot added a comment.Sep 27 2017, 9:45 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-27T09:45:13Z] <marostegui> Stop mysql on db1035 to copy its data to db1103 - T172679

gerritbot added a comment.Sep 27 2017, 9:48 AM

Change 380944 merged by jenkins-bot:
[operations/software@master] s3.hosts: Add db1103

https://gerrit.wikimedia.org/r/380944

gerritbot added a comment.Sep 27 2017, 12:13 PM

Change 380954 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1103 to the array

https://gerrit.wikimedia.org/r/380954

gerritbot added a comment.Sep 27 2017, 12:18 PM

Change 380954 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1103 to the array

https://gerrit.wikimedia.org/r/380954

Marostegui updated the task description. (Show Details)Sep 27 2017, 3:53 PM
Stashbot added a comment.Sep 27 2017, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2017-09-27T15:54:04Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1103 with low weight - T172679 (duration: 00m 47s)

Marostegui added a comment.Oct 2 2017, 9:10 AM
In T172679#3638670, @Marostegui wrote:
In T172679#3635468, @jcrespo wrote:

Yes, although we may need still an extra host for vlow/dumps, separate from the other services, and smaller in size. Any ideas about which one we can move that is relatively old but not about to be removed?

According to: https://gerrit.wikimedia.org/r/#/c/338996/1/wmf-config/db-eqiad.php

In s1 we'd have 2 of the most powerful servers serving API.
Currently we have 3x160G serving API, we could take one of those and get one of the big ones to serve API instead.

I am going to change the config (without taking any of the hosts away yet) and see how it would perform.
If it all goes fine, we can move one of the 160G to s3 vslow dumps (T172679#3635468) and decomm db1038

Marostegui added a parent task: T177208: Provide dedicated database resources for wikidata.Oct 2 2017, 12:45 PM
gerritbot added a comment.Oct 3 2017, 6:20 AM

Change 381929 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/381929

gerritbot added a comment.Oct 3 2017, 6:26 AM

Change 381929 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/381929

Stashbot added a comment.Oct 3 2017, 6:28 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-03T06:28:11Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1066 - T172679 (duration: 00m 47s)

Marostegui updated the task description. (Show Details)Oct 4 2017, 6:31 AM
gerritbot added a comment.Oct 4 2017, 6:33 AM

Change 382118 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087 to clone db1104

https://gerrit.wikimedia.org/r/382118

gerritbot added a comment.Oct 4 2017, 6:37 AM

Change 382118 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087 to clone db1104

https://gerrit.wikimedia.org/r/382118

Stashbot added a comment.Oct 4 2017, 6:39 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-04T06:39:08Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1087 - T172679 (duration: 00m 50s)

gerritbot added a comment.Oct 4 2017, 6:42 AM

Change 382119 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add db1104 to s5 cloned from db1087

https://gerrit.wikimedia.org/r/382119

gerritbot added a comment.Oct 4 2017, 6:45 AM

Change 382119 merged by Marostegui:
[operations/puppet@production] mariadb: Add db1104 to s5 cloned from db1087

https://gerrit.wikimedia.org/r/382119

gerritbot added a comment.Oct 4 2017, 6:51 AM

Change 382121 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s5.hosts: Add db1104 to s5

https://gerrit.wikimedia.org/r/382121

Stashbot added a comment.Oct 4 2017, 6:53 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-04T06:53:46Z] <marostegui> Stop MySQL on db1087 to clone db1104 from it - T172679

gerritbot added a comment.Oct 4 2017, 6:53 AM

Change 382121 merged by jenkins-bot:
[operations/software@master] s5.hosts: Add db1104 to s5

https://gerrit.wikimedia.org/r/382121

gerritbot added a comment.Oct 4 2017, 9:26 AM

Change 382141 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1104 to the config

https://gerrit.wikimedia.org/r/382141

gerritbot added a comment.Oct 4 2017, 9:30 AM

Change 382141 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1104 to the config

https://gerrit.wikimedia.org/r/382141

Stashbot added a comment.Oct 4 2017, 9:32 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-04T09:32:17Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db1104 to the config - T172679 (duration: 00m 51s)

Stashbot added a comment.Oct 4 2017, 9:33 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-04T09:33:13Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db1104 to the config - T172679 (duration: 00m 50s)

Marostegui updated the task description. (Show Details)Oct 4 2017, 9:33 AM
Marostegui updated the task description. (Show Details)Oct 5 2017, 6:28 AM
gerritbot added a comment.Oct 5 2017, 6:35 AM

Change 382372 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add db1105 to s5

https://gerrit.wikimedia.org/r/382372

gerritbot added a comment.Oct 5 2017, 6:36 AM

Change 382373 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s5.hosts: Add db1105

https://gerrit.wikimedia.org/r/382373

gerritbot added a comment.Oct 5 2017, 6:38 AM

Change 382372 merged by Marostegui:
[operations/puppet@production] mariadb: Add db1105 to s5

https://gerrit.wikimedia.org/r/382372

gerritbot added a comment.Oct 5 2017, 6:39 AM

Change 382373 merged by jenkins-bot:
[operations/software@master] s5.hosts: Add db1105

https://gerrit.wikimedia.org/r/382373

Stashbot added a comment.Oct 5 2017, 6:40 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-05T06:40:38Z] <marostegui> Stop MySQL on db1104 to clone db1105 from it - T172679

gerritbot added a comment.Oct 5 2017, 8:32 AM

Change 382388 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1105 to the config

https://gerrit.wikimedia.org/r/382388

gerritbot added a comment.Oct 5 2017, 8:38 AM

Change 382388 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1105 to the config

https://gerrit.wikimedia.org/r/382388

Stashbot added a comment.Oct 5 2017, 8:39 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-05T08:39:49Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db1105 to the config - T172679 (duration: 00m 52s)

Marostegui updated the task description. (Show Details)Oct 5 2017, 8:40 AM
Stashbot added a comment.Oct 5 2017, 8:40 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-05T08:40:45Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db1105 to the config - T172679 (duration: 00m 50s)

gerritbot added a comment.Oct 5 2017, 8:44 AM

Change 382391 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s5.hosts: Add db1106 to s5

https://gerrit.wikimedia.org/r/382391

gerritbot added a comment.Oct 5 2017, 8:46 AM

Change 382392 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add db1106 to s5

https://gerrit.wikimedia.org/r/382392

gerritbot added a comment.Oct 5 2017, 8:48 AM

Change 382391 merged by jenkins-bot:
[operations/software@master] s5.hosts: Add db1106 to s5

https://gerrit.wikimedia.org/r/382391

gerritbot added a comment.Oct 5 2017, 8:49 AM

Change 382392 merged by Marostegui:
[operations/puppet@production] mariadb: Add db1106 to s5

https://gerrit.wikimedia.org/r/382392

Stashbot added a comment.Oct 5 2017, 8:54 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-05T08:54:09Z] <marostegui> Stop MySQL on db1104 to clone db1106 - T172679

gerritbot added a comment.Oct 6 2017, 6:14 AM

Change 382648 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1106 to config

https://gerrit.wikimedia.org/r/382648

gerritbot added a comment.Oct 6 2017, 6:20 AM

Change 382648 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1106 to config

https://gerrit.wikimedia.org/r/382648

Stashbot added a comment.Oct 6 2017, 6:21 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-06T06:21:59Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db1106 to the config - T172679 (duration: 00m 47s)

Marostegui updated the task description. (Show Details)Oct 6 2017, 6:22 AM
Stashbot added a comment.Oct 6 2017, 6:23 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-06T06:23:32Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db1106 to the config - T172679 (duration: 00m 47s)

Marostegui closed this task as Resolved.Oct 6 2017, 6:23 AM
Marostegui claimed this task.

All these hosts have been productionized already.
I am going to close this task as resolved as they are now all in production. They might not be in they definitive place, but they are serving traffic anyways.

gerritbot added a comment.Oct 9 2017, 7:56 AM

Change 383076 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Move db1066 from s1 api to s3 vslow

https://gerrit.wikimedia.org/r/383076

Marostegui added a comment.Oct 9 2017, 7:57 AM
In T172679#3650005, @Marostegui wrote:
In T172679#3638670, @Marostegui wrote:
In T172679#3635468, @jcrespo wrote:

Yes, although we may need still an extra host for vlow/dumps, separate from the other services, and smaller in size. Any ideas about which one we can move that is relatively old but not about to be removed?

According to: https://gerrit.wikimedia.org/r/#/c/338996/1/wmf-config/db-eqiad.php

In s1 we'd have 2 of the most powerful servers serving API.
Currently we have 3x160G serving API, we could take one of those and get one of the big ones to serve API instead.

I am going to change the config (without taking any of the hosts away yet) and see how it would perform.
If it all goes fine, we can move one of the 160G to s3 vslow dumps (T172679#3635468) and decomm db1038

I am going to move db1066 from s1 api to s3 vslow service.
Once done, we can decommission db1038 which is becoming short on disk space.

gerritbot added a comment.Oct 9 2017, 8:02 AM

Change 383077 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1066 from s1 to s3

https://gerrit.wikimedia.org/r/383077

gerritbot added a comment.Oct 9 2017, 8:10 AM

Change 383076 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Move db1066 from s1 api to s3 vslow

https://gerrit.wikimedia.org/r/383076

Stashbot added a comment.Oct 9 2017, 8:12 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-09T08:12:23Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Move db1066 from s1 to s3 - T172679 (duration: 01m 25s)

Marostegui added a comment.Oct 9 2017, 8:17 AM

I am going to change my plans and move db1072 instead of db1066.
The reason for that change is that db1072 was cloned from db1052, so we are sure we still have its data on the master.
I have been tracing from where db1066 was cloned from and could not find it. So instead of dropping it for good without knowing its data status, I rather use db1072 which we know its data is somewhere.

gerritbot added a comment.Oct 9 2017, 8:20 AM

Change 383083 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Restore db1066 on s1, db1072 to s3

https://gerrit.wikimedia.org/r/383083

gerritbot added a comment.Oct 9 2017, 8:21 AM

Change 383077 abandoned by Marostegui:
mariadb: Move db1066 from s1 to s3

Reason:
Going to move db1072 instead

https://gerrit.wikimedia.org/r/383077

gerritbot added a comment.Oct 9 2017, 8:25 AM

Change 383083 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Restore db1066 on s1, db1072 to s3

https://gerrit.wikimedia.org/r/383083

Stashbot added a comment.Oct 9 2017, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-09T08:26:22Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1066 on s1 to and move db1072 to s3 instead - T172679 (duration: 00m 47s)

gerritbot added a comment.Oct 9 2017, 8:26 AM

Change 383085 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1072 to s3

https://gerrit.wikimedia.org/r/383085

Stashbot added a comment.Oct 10 2017, 6:05 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-10T06:05:55Z] <marostegui> Stop MySQL on db1072 to move it to s3 - T172679

gerritbot added a comment.Oct 10 2017, 6:10 AM

Change 383085 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1072 to s3

https://gerrit.wikimedia.org/r/383085

gerritbot added a comment.Oct 10 2017, 6:14 AM

Change 383309 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1038

https://gerrit.wikimedia.org/r/383309

gerritbot added a comment.Oct 10 2017, 6:18 AM

Change 383309 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1038

https://gerrit.wikimedia.org/r/383309

Stashbot added a comment.Oct 10 2017, 6:19 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-10T06:19:58Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1038 - T172679 (duration: 00m 47s)

Stashbot added a comment.Oct 10 2017, 6:22 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-10T06:22:07Z] <marostegui> Stop MySQL on db1038 to transfer its data to db1072 - T172679

gerritbot added a comment.Oct 10 2017, 6:30 AM

Change 383310 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s3.hosts,s1.hosts: Move db1072 from s1 to s3

https://gerrit.wikimedia.org/r/383310

gerritbot added a comment.Oct 10 2017, 6:32 AM

Change 383310 merged by jenkins-bot:
[operations/software@master] s3.hosts,s1.hosts: Move db1072 from s1 to s3

https://gerrit.wikimedia.org/r/383310

Stashbot added a comment.Oct 10 2017, 2:29 PM

Mentioned in SAL (#wikimedia-operations) [2017-10-10T14:29:16Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight - T172679 (duration: 00m 47s)

Stashbot added a comment.Oct 11 2017, 7:10 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-11T07:10:47Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1072 into the vslow and dump service for s3 - T172679 (duration: 00m 46s)

gerritbot added a comment.Oct 11 2017, 8:28 AM

Change 383516 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1103

https://gerrit.wikimedia.org/r/383516

gerritbot added a comment.Oct 11 2017, 8:36 AM

Change 383516 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1103

https://gerrit.wikimedia.org/r/383516

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL