Page MenuHomePhabricator
Create Task
Maniphest T183470

Setup newer machines and replace all old misc (m*) and x1 codfw machines
Closed, ResolvedPublic
Actions

Description

db2078 was already setup, although probably is too large to stay like that.
x1 host was setup, but there is not redundancy for it on codfw.

Pending to replace:

  • db2011
  • db2012
  • db2030

New hosts to setup:

  • s5 replica (or setup redundancy on eqiad only)

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Dec 21 2017, 3:06 PM
jcrespo triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 21 2017, 3:06 PM
jcrespo added a comment.Dec 21 2017, 3:07 PM

Eqiad version: T183469
Very related: T170662

jcrespo mentioned this in T170662: Productionize 22 new codfw database servers.Dec 21 2017, 3:08 PM
gerritbot subscribed.Dec 22 2017, 8:34 AM

Change 399782 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1055 & db1056 as x1 replicas

https://gerrit.wikimedia.org/r/399782

gerritbot added a project: Patch-For-Review.Dec 22 2017, 8:34 AM
Marostegui moved this task from Triage to In progress on the DBA board.Dec 27 2017, 7:24 AM
gerritbot added a comment.Jan 2 2018, 11:18 AM

Change 399782 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool db1055 & db1056 as x1 replicas

https://gerrit.wikimedia.org/r/399782

Marostegui moved this task from In progress to Pending comment on the DBA board.Feb 9 2018, 1:16 PM
gerritbot added a comment.Feb 15 2018, 10:30 AM

Change 410790 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Remove db2042 from mediawiki

https://gerrit.wikimedia.org/r/410790

gerritbot added a comment.Feb 15 2018, 10:48 AM

Change 410794 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Move db2042 from codfw:core:s1 to misc:s3 (phabricator)

https://gerrit.wikimedia.org/r/410794

gerritbot added a comment.Feb 15 2018, 2:18 PM

Change 410790 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Remove db2042 from mediawiki

https://gerrit.wikimedia.org/r/410790

gerritbot added a comment.Feb 16 2018, 10:18 AM

Change 411198 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Prepare reimage of db1053 and db2042 to strech

https://gerrit.wikimedia.org/r/411198

gerritbot added a comment.Feb 16 2018, 10:24 AM

Change 411198 merged by Jcrespo:
[operations/puppet@production] mariadb: Prepare reimage of db1053 and db2042 to strech

https://gerrit.wikimedia.org/r/411198

ops-monitoring-bot subscribed.Feb 16 2018, 10:34 AM

Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts:

['db2042.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201802161034_jynus_12767.log.

gerritbot added a comment.Feb 16 2018, 10:36 AM

Change 410794 merged by Jcrespo:
[operations/puppet@production] mariadb: Move db2042 from codfw:core:s1 to misc:s3 (phabricator)

https://gerrit.wikimedia.org/r/410794

ops-monitoring-bot added a comment.Feb 16 2018, 10:59 AM

Completed auto-reimage of hosts:

['db2042.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Feb 16 2018, 11:05 AM

Change 411201 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Disable notifications for db1043 and db2012

https://gerrit.wikimedia.org/r/411201

gerritbot added a comment.Feb 16 2018, 11:06 AM

Change 411201 merged by Jcrespo:
[operations/puppet@production] mariadb: Disable notifications for db1043 and db2012

https://gerrit.wikimedia.org/r/411201

gerritbot added a comment.Feb 16 2018, 11:21 AM

Change 411203 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Change phabricator db config to modern defaults

https://gerrit.wikimedia.org/r/411203

gerritbot added a comment.Feb 16 2018, 11:33 AM

Change 411203 merged by Jcrespo:
[operations/puppet@production] mariadb: Change phabricator db config to modern defaults

https://gerrit.wikimedia.org/r/411203

gerritbot added a comment.Feb 16 2018, 12:21 PM

Change 411216 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Mariadb: Set default basedir for phabricator (Mariadb 10.1)

https://gerrit.wikimedia.org/r/411216

gerritbot added a comment.Feb 16 2018, 12:23 PM

Change 411216 merged by Jcrespo:
[operations/puppet@production] Mariadb: Set default basedir for phabricator (Mariadb 10.1)

https://gerrit.wikimedia.org/r/411216

jcrespo added a parent task: T176243: Decommission database hosts <= db2031 (tracking).Feb 20 2018, 7:04 PM
jcrespo removed a parent task: T134476: Decommission old coredb machines (<=db1050).
gerritbot added a comment.Feb 20 2018, 7:05 PM

Change 412978 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Allow db2044 reimage on stretch

https://gerrit.wikimedia.org/r/412978

gerritbot added a comment.Feb 20 2018, 7:08 PM

Change 412978 merged by Jcrespo:
[operations/puppet@production] mariadb: Allow db2044 reimage on stretch

https://gerrit.wikimedia.org/r/412978

jcrespo added a subtask: T187722: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030).Feb 20 2018, 8:01 PM
jcrespo updated the task description. (Show Details)
gerritbot added a comment.Feb 20 2018, 8:07 PM

Change 412994 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Move db2044 from codfw-core-s4 to codfw-misc-s2

https://gerrit.wikimedia.org/r/412994

gerritbot added a comment.Feb 20 2018, 8:10 PM

Change 412995 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Setup db2044 as the main m2 host on codfw and monitor it

https://gerrit.wikimedia.org/r/412995

jcrespo added a parent task: T170662: Productionize 22 new codfw database servers.Feb 21 2018, 9:41 AM
ops-monitoring-bot added a comment.Feb 21 2018, 10:42 AM

Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts:

['db2044.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201802211042_jynus_5930.log.

gerritbot added a comment.Feb 21 2018, 10:44 AM

Change 412994 merged by Jcrespo:
[operations/puppet@production] mariadb: Move db2044 from codfw-core-s4 to codfw-misc-s2

https://gerrit.wikimedia.org/r/412994

ops-monitoring-bot added a comment.Feb 21 2018, 11:10 AM

Completed auto-reimage of hosts:

['db2044.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Feb 21 2018, 11:32 AM

Change 412995 merged by Jcrespo:
[operations/puppet@production] dbproxy: Setup db2044 as the main m2 host on codfw and monitor it

https://gerrit.wikimedia.org/r/412995

gerritbot added a comment.Feb 21 2018, 1:44 PM

Change 413159 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Reenable notifications on db2044 (m2)

https://gerrit.wikimedia.org/r/413159

gerritbot added a comment.Feb 21 2018, 2:38 PM

Change 413167 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Update socket location of misc services (m1, m2, m5)

https://gerrit.wikimedia.org/r/413167

gerritbot added a comment.Feb 21 2018, 2:46 PM

Change 413159 merged by Jcrespo:
[operations/puppet@production] mariadb: Reenable notifications on db2044 (m2)

https://gerrit.wikimedia.org/r/413159

jcrespo added a comment.Feb 21 2018, 3:28 PM

I would say this is resolved- only pending the actual decommission (tracked on separate tickets), and the setup of extra servers for redundancy (tracked on T170662).

jcrespo closed this task as Resolved.Feb 21 2018, 3:30 PM
jcrespo claimed this task.
Marostegui closed subtask T187722: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) as Resolved.Feb 22 2018, 5:08 PM
jcrespo added a parent task: T157702: Followup for TLS MariaDB server roll-out.Mar 9 2018, 3:30 PM
gerritbot added a comment.Mar 28 2018, 2:28 PM

Change 413167 abandoned by Jcrespo:
mariadb: Update socket location of misc services (m1, m2, m5)

Reason:
Already merged elsewhere.

https://gerrit.wikimedia.org/r/413167

Papaul closed subtask T187543: Decommission db2012 as Resolved.Apr 18 2018, 1:10 AM
RobH closed subtask T187886: Decommission db2011 as Resolved.May 3 2018, 3:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits