Page MenuHomePhabricator

Switchover codfw primary database masters to new hosts
Closed, ResolvedPublic

Description

All the codfw old masters need to be switched over to the new hosts (512GB+SSDs)
List of shards

  • s1 db2048 -> db2112
  • s2 db2035 -> db2107
  • s3 db2043 -> db2105
  • s4 db2051 -> db2090
  • s5 db2052 -> db2123
  • s6 db2046 -> db2129
  • s7 db2047 -> db2118
  • s8 db2045 -> db2079
  • x1 db2069 -> db2096

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 529704 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2105 to s3 codfw master

https://gerrit.wikimedia.org/r/529704

Mentioned in SAL (#wikimedia-operations) [2019-08-12T07:43:15Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db2105 to s3 codfw master T230106', diff saved to https://phabricator.wikimedia.org/P8900 and previous config saved to /var/cache/conftool/dbconfig/20190812-074314-marostegui.json

Change 529705 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Re-organize s3 codfw

https://gerrit.wikimedia.org/r/529705

Change 529725 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2127: Clarify it is a candidate master

https://gerrit.wikimedia.org/r/529725

Change 529725 merged by Marostegui:
[operations/puppet@production] db2127: Clarify it is a candidate master

https://gerrit.wikimedia.org/r/529725

Marostegui updated the task description. (Show Details)Aug 12 2019, 9:57 AM

Change 531049 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Re-organize s5 codfw

https://gerrit.wikimedia.org/r/531049

Change 531049 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Re-organize s5 codfw

https://gerrit.wikimedia.org/r/531049

Mentioned in SAL (#wikimedia-operations) [2019-08-20T07:50:12Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Reorganize s5 codfw weights T230106 (duration: 00m 47s)

Change 531050 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2123 to s5 codfw master

https://gerrit.wikimedia.org/r/531050

Change 531051 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Promote db2123 to s5 codfw master

https://gerrit.wikimedia.org/r/531051

Mentioned in SAL (#wikimedia-operations) [2019-08-20T08:05:52Z] <marostegui> Switchover s5 codfw master db2052 -> db2123 T230106

Change 531050 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2123 to s5 codfw master

https://gerrit.wikimedia.org/r/531050

Change 531051 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Promote db2123 to s5 codfw master

https://gerrit.wikimedia.org/r/531051

Mentioned in SAL (#wikimedia-operations) [2019-08-20T08:19:57Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2123 to s5 codfw master T230106 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2019-08-20T08:24:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Re-organize s5 codfw weights - T230106', diff saved to https://phabricator.wikimedia.org/P8935 and previous config saved to /var/cache/conftool/dbconfig/20190820-082411-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-08-20T08:28:03Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db2123 as codfw s5 master - T230106', diff saved to https://phabricator.wikimedia.org/P8936 and previous config saved to /var/cache/conftool/dbconfig/20190820-082802-marostegui.json

Marostegui updated the task description. (Show Details)Aug 20 2019, 8:35 AM

s5 codfw master has been swapped from db2052 to db2123

Change 532370 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2114: Change binlog format to STATEMENT

https://gerrit.wikimedia.org/r/532370

Change 532371 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Specify db2114 status

https://gerrit.wikimedia.org/r/532371

Change 532370 merged by Marostegui:
[operations/puppet@production] db2114: Change binlog format to STATEMENT

https://gerrit.wikimedia.org/r/532370

Change 532371 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Specify db2114 status

https://gerrit.wikimedia.org/r/532371

Marostegui updated the task description. (Show Details)Aug 27 2019, 5:55 AM

Change 532518 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2129 as codfw master for s6

https://gerrit.wikimedia.org/r/532518

Change 532543 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Promote db2129 to s6 codfw master

https://gerrit.wikimedia.org/r/532543

Mentioned in SAL (#wikimedia-operations) [2019-08-27T07:14:56Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db2129 weight to 0 before promoting it to codfw s6 master T230106', diff saved to https://phabricator.wikimedia.org/P8980 and previous config saved to /var/cache/conftool/dbconfig/20190827-071456-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-08-27T07:16:37Z] <marostegui> Switchover codfw s6 master from db2046 to db2129 T230106

Change 532518 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2129 as codfw master for s6

https://gerrit.wikimedia.org/r/532518

Mentioned in SAL (#wikimedia-operations) [2019-08-27T07:25:57Z] <marostegui@cumin1001> dbctl commit (dc=codfw): 'Promote db2129 to codfw s6 master T230106', diff saved to https://phabricator.wikimedia.org/P8981 and previous config saved to /var/cache/conftool/dbconfig/20190827-072556-marostegui.json

Change 532543 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Promote db2129 to s6 codfw master

https://gerrit.wikimedia.org/r/532543

Mentioned in SAL (#wikimedia-operations) [2019-08-27T07:28:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2046, this host will be decommissioned T230106', diff saved to https://phabricator.wikimedia.org/P8982 and previous config saved to /var/cache/conftool/dbconfig/20190827-072847-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-08-27T07:28:57Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2129 as s6 codfw master T230106 (duration: 00m 46s)

Change 532548 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Re-organize s6 codfw

https://gerrit.wikimedia.org/r/532548

Mentioned in SAL (#wikimedia-operations) [2019-08-27T07:48:02Z] <marostegui@cumin1001> dbctl commit (dc=codfw): 'Reorganize s6 codfw weights and roles T230106', diff saved to https://phabricator.wikimedia.org/P8983 and previous config saved to /var/cache/conftool/dbconfig/20190827-074802-marostegui.json

Change 532548 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Re-organize s6 codfw

https://gerrit.wikimedia.org/r/532548

Mentioned in SAL (#wikimedia-operations) [2019-08-27T07:50:22Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Reorganize s6 codfw weights and roles T230106 (duration: 00m 44s)

Marostegui updated the task description. (Show Details)Aug 27 2019, 8:09 AM

s6 codfw master swapped from db2046 to db2129

Mentioned in SAL (#wikimedia-operations) [2019-08-28T06:41:43Z] <marostegui> Upgrade mysql on s7 codfw hosts: db2054, db2061, db2068, db2077 - T230106

Marostegui updated the task description. (Show Details)Sep 3 2019, 4:58 AM

Change 533989 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2118 to s7 codfw master

https://gerrit.wikimedia.org/r/533989

Change 533990 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Promote db2118 to s7 codfw master

https://gerrit.wikimedia.org/r/533990

Mentioned in SAL (#wikimedia-operations) [2019-09-03T05:02:18Z] <marostegui> Promote db2118 to s7 codfw master (db2047 -> db2118) T230106

Change 533989 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2118 to s7 codfw master

https://gerrit.wikimedia.org/r/533989

Change 533990 merged by Marostegui:
[operations/mediawiki-config@master] db-codfw.php: Promote db2118 to s7 codfw master

https://gerrit.wikimedia.org/r/533990

Mentioned in SAL (#wikimedia-operations) [2019-09-03T05:14:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db2118 to s7 codfw master (db2047 -> db2118) T230106', diff saved to https://phabricator.wikimedia.org/P9026 and previous config saved to /var/cache/conftool/dbconfig/20190903-051450-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-03T05:16:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2047 old master from s7 T230106', diff saved to https://phabricator.wikimedia.org/P9027 and previous config saved to /var/cache/conftool/dbconfig/20190903-051619-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-03T05:17:09Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2118 to s7 codfw master (db2047 -> db2118) T230106 (duration: 00m 54s)

Change 533993 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Reorganize s7 codfw

https://gerrit.wikimedia.org/r/533993

Change 533993 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Reorganize s7 codfw

https://gerrit.wikimedia.org/r/533993

Mentioned in SAL (#wikimedia-operations) [2019-09-03T05:47:49Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Reorganize s7 codfw T230106 (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-09-03T05:52:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Reorganize s7 codfw T230106', diff saved to https://phabricator.wikimedia.org/P9028 and previous config saved to /var/cache/conftool/dbconfig/20190903-055234-marostegui.json

s7 codfw master swapped from db2047 to db2118

Marostegui updated the task description. (Show Details)Sep 3 2019, 5:53 AM

Change 534014 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Re-organize s1 codfw candidate masters

https://gerrit.wikimedia.org/r/534014

Change 534014 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Re-organize s1 codfw candidate masters

https://gerrit.wikimedia.org/r/534014

Change 534018 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Reorganize future master/candidate on s1 codfw

https://gerrit.wikimedia.org/r/534018

Change 534018 merged by Marostegui:
[operations/puppet@production] mariadb: Reorganize future master/candidate on s1 codfw

https://gerrit.wikimedia.org/r/534018

Marostegui updated the task description. (Show Details)Sep 11 2019, 5:15 AM

Change 535764 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2112 to s1 codfw master

https://gerrit.wikimedia.org/r/535764

Mentioned in SAL (#wikimedia-operations) [2019-09-11T05:29:01Z] <marostegui> Switchover s1 codfw master db2048 -> db2112 T230106

Change 535764 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2112 to s1 codfw master

https://gerrit.wikimedia.org/r/535764

Mentioned in SAL (#wikimedia-operations) [2019-09-11T05:47:54Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db2112 to s1 codfw master T230106', diff saved to https://phabricator.wikimedia.org/P9079 and previous config saved to /var/cache/conftool/dbconfig/20190911-054753-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-11T05:48:56Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2048, will be decommissioned T230106', diff saved to https://phabricator.wikimedia.org/P9080 and previous config saved to /var/cache/conftool/dbconfig/20190911-054855-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-11T06:17:00Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9081 and previous config saved to /var/cache/conftool/dbconfig/20190911-061659-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-11T06:19:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9082 and previous config saved to /var/cache/conftool/dbconfig/20190911-061924-marostegui.json

Marostegui closed this task as Resolved.Sep 11 2019, 9:38 AM
Marostegui updated the task description. (Show Details)

This is all done