Page MenuHomePhabricator
Create Task
Maniphest T170662

Productionize 22 new codfw database servers
Closed, ResolvedPublic
Actions

Description

More than a task, this is a tracking list to not "lose" not yet in-production servers and coordinate how to set them up.

These servers T162159 have to be used to:

  • Decom servers <db2030
  • Setup misc servers appropiately
  • Setup the eventual s8

State:

  • db2071: pooled as s1 api
  • db2072: pooled into s1
  • db2073: pooled in s4
  • db2074: pooled in s3
  • db2075: pooled in s5
  • db2076: pooled in s6
  • db2077: pooled in s7
  • db2078: pooled in m1 (probably too large for that, we may change it in the future or add more shards)
  • db2079: provisioned on s5 and later s8
  • db2080: provisioned on s5 and later s8
  • db2081: provisioned on s5 and later s8
  • db2082: provisioned on s5 and later s8
  • db2083: provisioned on s5 and later s8
  • db2084: provisioned on s5 and later s8 and used to test multi-instance (T178359)
  • db2085: provisioned on s1, s8 as recentchanges T178359
  • db2086: provisioned on s5, s7 as recentchanges T178359
  • db2087: provisioned on s6, s7 as recentchanges T178359
  • db2088: provisioned on s1, s2 as recentchanges T178359
  • db2089: provisioned on s6, s8 as recentchanges T178359
  • db2090: provisioned on s4 in exchange for 2 servers sent to misc
  • db2091: provisioned on s2, s4 as recentchanges T178359
  • db2092: provisioned on s1 as api T191275#4099560

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Oct 17 2017, 7:53 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-17T07:53:40Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db2086 to the config - T170662 (duration: 00m 45s)

Marostegui updated the task description. (Show Details)Oct 17 2017, 10:08 AM
Marostegui updated the task description. (Show Details)Oct 19 2017, 6:07 AM
gerritbot added a comment.Oct 19 2017, 6:20 AM

Change 385135 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add db2092 to s1 and s3 as rc

https://gerrit.wikimedia.org/r/385135

gerritbot added a comment.Oct 19 2017, 6:24 AM

Change 385135 merged by Marostegui:
[operations/puppet@production] mariadb: Add db2092 to s1 and s3 as rc

https://gerrit.wikimedia.org/r/385135

Marostegui updated the task description. (Show Details)Oct 19 2017, 12:33 PM
Marostegui updated the task description. (Show Details)Oct 24 2017, 8:34 AM
gerritbot added a comment.Oct 24 2017, 8:35 AM

Change 386135 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2088 with stretch

https://gerrit.wikimedia.org/r/386135

gerritbot added a comment.Oct 24 2017, 8:43 AM

Change 386135 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2088 with stretch

https://gerrit.wikimedia.org/r/386135

ops-monitoring-bot added a subscriber: ops-monitoring-bot.Oct 24 2017, 8:48 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db2088.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201710240848_marostegui_19799_db2088_codfw_wmnet.log.

ops-monitoring-bot added a comment.Oct 24 2017, 9:07 AM

Completed auto-reimage of hosts:

['db2088.codfw.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Oct 25 2017, 10:35 AM
gerritbot added a comment.Oct 25 2017, 10:37 AM

Change 386357 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2091 as stretch

https://gerrit.wikimedia.org/r/386357

Marostegui updated the task description. (Show Details)Oct 25 2017, 10:37 AM
gerritbot added a comment.Oct 25 2017, 10:38 AM

Change 386357 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2091 as stretch

https://gerrit.wikimedia.org/r/386357

ops-monitoring-bot added a comment.Oct 25 2017, 10:41 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db2091.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201710251040_marostegui_20773_db2091_codfw_wmnet.log.

ops-monitoring-bot added a comment.Oct 25 2017, 10:59 AM

Completed auto-reimage of hosts:

['db2091.codfw.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Oct 26 2017, 12:37 PM
Marostegui updated the task description. (Show Details)Oct 26 2017, 12:54 PM
Marostegui updated the task description. (Show Details)Oct 30 2017, 7:38 AM
Marostegui updated the task description. (Show Details)Oct 30 2017, 8:03 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Oct 30 2017, 11:17 AM
gerritbot added a comment.Oct 31 2017, 6:43 AM

Change 387524 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2089 with stretch

https://gerrit.wikimedia.org/r/387524

Marostegui updated the task description. (Show Details)Oct 31 2017, 6:43 AM
gerritbot added a comment.Oct 31 2017, 6:44 AM

Change 387524 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2089 with stretch

https://gerrit.wikimedia.org/r/387524

Marostegui updated the task description. (Show Details)Oct 31 2017, 11:19 AM
Marostegui updated the task description. (Show Details)Oct 31 2017, 2:12 PM
Marostegui updated the task description. (Show Details)Nov 3 2017, 2:01 PM
gerritbot added a comment.Nov 20 2017, 10:11 AM

Change 392400 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Leave reimaginable only the db latest servers

https://gerrit.wikimedia.org/r/392400

Marostegui moved this task from In progress to Next on the DBA board.Nov 21 2017, 10:22 AM
gerritbot added a comment.Nov 27 2017, 10:22 AM

Change 392400 merged by Jcrespo:
[operations/puppet@production] mariadb: Leave reimaginable only the db latest servers

https://gerrit.wikimedia.org/r/392400

gerritbot added a comment.Dec 1 2017, 2:31 PM

Change 394573 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Undeploy db2092, use db1085 for s1 (remove s3 special slaves)

https://gerrit.wikimedia.org/r/394573

gerritbot added a comment.Dec 1 2017, 3:03 PM

Change 394573 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Undeploy db2092, use db2085 for s1 (remove s3 special slaves)

https://gerrit.wikimedia.org/r/394573

gerritbot added a comment.Dec 1 2017, 3:39 PM

Change 394588 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Set db2085 with s1 and s8; make db2092 pending to be deployed

https://gerrit.wikimedia.org/r/394588

jcrespo updated the task description. (Show Details)Dec 1 2017, 3:48 PM
gerritbot added a comment.Dec 1 2017, 3:51 PM

Change 394588 merged by Jcrespo:
[operations/puppet@production] Set db2085 with s1 and s8; make db2092 pending to be deployed

https://gerrit.wikimedia.org/r/394588

gerritbot added a comment.Dec 1 2017, 3:51 PM

Change 394591 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] Remove db2092; setup db1085 with s1 and s8

https://gerrit.wikimedia.org/r/394591

gerritbot added a comment.Dec 1 2017, 3:53 PM

Change 394591 merged by Jcrespo:
[operations/software@master] Remove db2092; setup db1085 with s1 and s8

https://gerrit.wikimedia.org/r/394591

gerritbot added a comment.Dec 1 2017, 4:10 PM

Change 394596 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Set db2092 as spare explicity

https://gerrit.wikimedia.org/r/394596

gerritbot added a comment.Dec 1 2017, 4:12 PM

Change 394596 merged by Jcrespo:
[operations/puppet@production] mariadb: Set db2092 as spare explicity

https://gerrit.wikimedia.org/r/394596

jcrespo added a comment.Dec 1 2017, 6:15 PM

Latest updates: T178359#3803600 , reflected on the description already.

Marostegui removed Marostegui as the assignee of this task.Jan 4 2018, 7:24 AM
gerritbot added a comment.Feb 22 2018, 9:53 AM

Change 413328 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] install_server: Set db2090, db2073 to reimage as stretch

https://gerrit.wikimedia.org/r/413328

gerritbot added a comment.Feb 22 2018, 10:07 AM

Change 413332 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db2073 for maintenance

https://gerrit.wikimedia.org/r/413332

gerritbot added a comment.Feb 22 2018, 10:07 AM

Change 413328 merged by Jcrespo:
[operations/puppet@production] install_server: Set db2090, db2073 to reimage as stretch

https://gerrit.wikimedia.org/r/413328

gerritbot added a comment.Feb 22 2018, 10:10 AM

Change 413332 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Depool db2073 for maintenance

https://gerrit.wikimedia.org/r/413332

gerritbot added a comment.Feb 22 2018, 10:15 AM

Change 413335 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Add db2090 to s4 section

https://gerrit.wikimedia.org/r/413335

gerritbot added a comment.Feb 22 2018, 10:22 AM

Change 413335 merged by Jcrespo:
[operations/puppet@production] mariadb: Add db2090 to s4 section

https://gerrit.wikimedia.org/r/413335

ops-monitoring-bot added a comment.Feb 22 2018, 10:24 AM

Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts:

['db2090.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201802221024_jynus_4608.log.

gerritbot added a comment.Feb 22 2018, 10:28 AM

Change 413337 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: disable alerts on db2073, db2090 and assign them to s4

https://gerrit.wikimedia.org/r/413337

gerritbot added a comment.Feb 22 2018, 10:30 AM

Change 413337 merged by Jcrespo:
[operations/puppet@production] mariadb: disable alerts on db2073, db2090 and assign them to s4

https://gerrit.wikimedia.org/r/413337

ops-monitoring-bot added a comment.Feb 22 2018, 10:44 AM

Completed auto-reimage of hosts:

['db2090.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2018, 12:21 PM

Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts:

['db2073.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201802221220_jynus_30806.log.

ops-monitoring-bot added a comment.Feb 22 2018, 12:43 PM

Completed auto-reimage of hosts:

['db2073.codfw.wmnet']

and were ALL successful.

jcrespo updated the task description. (Show Details)Feb 22 2018, 7:08 PM
gerritbot added a comment.Feb 22 2018, 7:09 PM

Change 413439 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool db2090 for the first time on s4

https://gerrit.wikimedia.org/r/413439

gerritbot added a comment.Feb 23 2018, 10:05 AM

Change 413439 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Pool db2090 for the first time on s4

https://gerrit.wikimedia.org/r/413439

gerritbot added a comment.Mar 8 2018, 9:42 AM

Change 417218 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s4.hosts: Add db2090

https://gerrit.wikimedia.org/r/417218

gerritbot added a comment.Mar 8 2018, 9:44 AM

Change 417218 merged by jenkins-bot:
[operations/software@master] s4.hosts: Add db2090

https://gerrit.wikimedia.org/r/417218

gerritbot added a comment.Apr 9 2018, 6:29 AM

Change 424994 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db2092 to s1

https://gerrit.wikimedia.org/r/424994

gerritbot added a comment.Apr 9 2018, 6:33 AM

Change 424997 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2072

https://gerrit.wikimedia.org/r/424997

gerritbot added a comment.Apr 9 2018, 6:34 AM

Change 424994 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2092 to s1

https://gerrit.wikimedia.org/r/424994

gerritbot added a comment.Apr 9 2018, 6:36 AM

Change 424997 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2072

https://gerrit.wikimedia.org/r/424997

Stashbot added a comment.Apr 9 2018, 6:38 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-09T06:38:01Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2072 - T170662 (duration: 00m 59s)

Stashbot added a comment.Apr 9 2018, 6:41 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-09T06:41:01Z] <marostegui> Stop MySQL on db2072 to clone db2092 from it - T170662

ops-monitoring-bot added a comment.Apr 9 2018, 6:50 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2092.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804090650_marostegui_30453.log.

gerritbot added a comment.Apr 9 2018, 7:15 AM

Change 425001 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reinstall db2092

https://gerrit.wikimedia.org/r/425001

gerritbot added a comment.Apr 9 2018, 7:17 AM

Change 425001 merged by Marostegui:
[operations/puppet@production] install_server: Allow reinstall db2092

https://gerrit.wikimedia.org/r/425001

ops-monitoring-bot added a comment.Apr 9 2018, 7:27 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2092.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201804090727_marostegui_6390.log.

ops-monitoring-bot added a comment.Apr 9 2018, 7:48 AM

Completed auto-reimage of hosts:

['db2092.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Apr 9 2018, 8:55 AM

Change 425013 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s1.hosts: Add db2092 to s1

https://gerrit.wikimedia.org/r/425013

gerritbot added a comment.Apr 9 2018, 8:57 AM

Change 425013 merged by jenkins-bot:
[operations/software@master] s1.hosts: Add db2092 to s1

https://gerrit.wikimedia.org/r/425013

gerritbot added a comment.Apr 9 2018, 8:58 AM

Change 425014 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db2092 to the config

https://gerrit.wikimedia.org/r/425014

gerritbot added a comment.Apr 9 2018, 9:00 AM

Change 425014 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db2092 to the config

https://gerrit.wikimedia.org/r/425014

Stashbot added a comment.Apr 9 2018, 9:02 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-09T09:02:22Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db2092 to the config - T170662 (duration: 00m 58s)

Stashbot added a comment.Apr 9 2018, 9:03 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-09T09:03:38Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db2092 to the config - T170662 (duration: 00m 59s)

gerritbot added a comment.Apr 9 2018, 10:00 AM

Change 425022 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2092.yaml: Enable notifications

https://gerrit.wikimedia.org/r/425022

gerritbot added a comment.Apr 9 2018, 10:01 AM

Change 425022 merged by Marostegui:
[operations/puppet@production] db2092.yaml: Enable notifications

https://gerrit.wikimedia.org/r/425022

gerritbot added a comment.Apr 9 2018, 1:07 PM

Change 425047 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Pool db2092 in s1

https://gerrit.wikimedia.org/r/425047

gerritbot added a comment.Apr 9 2018, 1:50 PM

Change 425047 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Pool db2092 in s1

https://gerrit.wikimedia.org/r/425047

Stashbot added a comment.Apr 9 2018, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-09T13:52:46Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Pool db2092 in s1 T170662 (duration: 00m 59s)

Marostegui closed this task as Resolved.Apr 9 2018, 1:54 PM
Marostegui updated the task description. (Show Details)

All the hosts are now pooled.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL