Page MenuHomePhabricator
Create Task
Maniphest T220170

Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment
Closed, ResolvedPublic
Actions

Description

Tracking task for the Q4 goal for 2018

  • Rack and setup 13 eqiad hosts T211613
  • Productionize 13 eqiad hosts T222682
  • Rack and setup 18 codfw hosts T221532
  • Productionize 18 codfw hosts T222772
    • Mimic eqiad db-eqiad to db-codfw as much as possible (while we still have to get the pending 10 hosts)

Details

SubjectRepoBranchLines +/-
db-codfw.php: Reorganize s3 codfw weightsoperations/mediawiki-configmaster+6 -6
mariadb: Promote db2096 to x1 codfw masteroperations/puppetproduction+3 -3
db-codfw.php: Promote db2096 to x1 codfw masteroperations/mediawiki-configmaster+2 -2
db2105: Change binlog formatoperations/puppetproduction+1 -0
mariadb: Promote db2107 to codfw s2 masteroperations/puppetproduction+3 -3
mariadb: db2051 disable notificationsoperations/puppetproduction+2 -0
db-codfw.php: Promote db2090 to s4 codfw masteroperations/mediawiki-configmaster+3 -3
mariadb: Promote db2090 to codfw s4 masteroperations/puppetproduction+5 -4
db-codfw.php: Mimic codfw weights to eqiadoperations/mediawiki-configmaster+5 -5
db2058,db2065: Enable notificationsoperations/puppetproduction+0 -2
db-eqiad,db-codfw.php: Remove db2062operations/mediawiki-configmaster+0 -4
mariadb: Move db2062 from s1 to m1operations/puppetproduction+11 -4
db-codfw.php: Clarify db2017 statusoperations/mediawiki-configmaster+1 -1
db2107: Change binlog formatoperations/puppetproduction+1 -0
db-codfw.php: Tackle s8 weightsoperations/mediawiki-configmaster+2 -2
db-codfw.php: Tackle s4 weightsoperations/mediawiki-configmaster+13 -14
db2090: Make it candidate master for s4operations/puppetproduction+1 -0
db-codfw.php: Repool db2104operations/mediawiki-configmaster+2 -1
db-codfw.php: Reorganize s8operations/mediawiki-configmaster+9 -9
mariadb: Make db2080 candidate master for s8 codfwoperations/puppetproduction+1 -1
db-codfw.php: Promote db2079 to s8 codfw master.operations/mediawiki-configmaster+5 -5
mariadb: Promote db2079 to s8 codfw masteroperations/puppetproduction+5 -5
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedaaronT88445 MediaWiki active/active datacenter investigation and work (tracking)
 ResolvedMarosteguiT220170 Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment
 ResolvedMarosteguiT211613 rack/setup/install db11[26-38].eqiad.wmnet
 ResolvedMarosteguiT222682 Productionize db11[26-38]
 Resolved• CmjohnsonT222731 Storage problems with new host db1133
Unknown Object (Task)
 ResolvedPapaulT221532 rack/setup/install db2[103-120].codfw.wmnet (18 hosts)
 ResolvedMarosteguiT222772 Productionize db2[103-120]
 ResolvedPapaulT222845 db2112 doesn't show service tag in idrac
 ResolvedMarosteguiT224852 Failover s4 primary master: db1068 to db1081
 ResolvedJohanT224516 Database primary master failover on s4 (commonswiki)
Unknown Object (Task)
 ResolvedMarosteguiT227113 rack/setup/install db21[21-30].codfw.wmnet
 ResolvedMarosteguiT226358 Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC
 ResolvedMarosteguiT227062 Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required)
 ResolvedJohanT227063 Database primary master failover on s8 (wikidatawiki)
 ResolvedMarosteguiT226952 Failover m2 master db1065 to db1132
Unknown Object (Task)
 ResolvedMarosteguiT229251 (2019-08-31)rack/setup/install db2131.codfw.wmnet
 ResolvedMarosteguiT228969 Productionize db21[21-31}
 ResolvedMarosteguiT228243 Switchover m3 (phabricator) master db1072 to db1128
 Resolved• mmodellT228955 test
 ResolvedMarosteguiT229657 Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC
 ResolvedMarosteguiT221533 Decommission old coredb machines (<=db2042)
 ResolvedMarosteguiT219493 Prepare to decommission 2 codfw x1 hosts db2033 and db2034
 ResolvedPapaulT220070 Decommission db2033
 ResolvedPapaulT223216 Decommission db2034
 ResolvedPapaulT223885 Decommission db2036
 ResolvedPapaulT223950 Decommission db2041
 ResolvedPapaulT224079 Decommission db2040
 ResolvedPapaulT224720 Decommission db2037
 ResolvedPapaulT225090 Decommission db2042
 ResolvedjbondT225278 Installation failing on late_command.sh
ResolvedRequestPapaulT225988 decommission db2039
 ResolvedRequestPapaulT227565 decommission db2038
 ResolvedPapaulT229784 Decommission db2035
 ResolvedMarosteguiT230106 Switchover codfw primary database masters to new hosts
 DeclinedMarosteguiT230459 Replace db2044 with db2063
Unknown Object (Task)
 ResolvedPapaulT237702 (codfw):rack/setup/install db213[2-5]
 ResolvedMarosteguiT238183 Productionize db213[2-5}
 ResolvedPapaulT233185 Decommission db2067.codfw.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.May 22 2019, 7:34 AM

Change 511818 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Tackle s8 weights

https://gerrit.wikimedia.org/r/511818

gerritbot added a comment.May 22 2019, 7:40 AM

Change 511818 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Tackle s8 weights

https://gerrit.wikimedia.org/r/511818

Stashbot added a comment.May 22 2019, 7:41 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-22T07:41:18Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Tackle s8 codfw weights T220170 (duration: 00m 55s)

Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:32 PM
gerritbot added a comment.May 23 2019, 5:13 AM

Change 512091 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2107

https://gerrit.wikimedia.org/r/512091

gerritbot added a project: Patch-For-Review.May 23 2019, 5:13 AM
gerritbot added a comment.May 23 2019, 5:15 AM

Change 512092 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2107: Change binlog format

https://gerrit.wikimedia.org/r/512092

gerritbot added a comment.May 23 2019, 5:16 AM

Change 512092 merged by Marostegui:
[operations/puppet@production] db2107: Change binlog format

https://gerrit.wikimedia.org/r/512092

gerritbot added a comment.May 23 2019, 5:19 AM

Change 512091 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Clarify db2017 status

https://gerrit.wikimedia.org/r/512091

Maintenance_bot removed a project: Patch-For-Review.May 23 2019, 10:45 AM
gerritbot added a comment.May 24 2019, 5:27 AM

Change 512311 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2062

https://gerrit.wikimedia.org/r/512311

gerritbot added a project: Patch-For-Review.May 24 2019, 5:27 AM
gerritbot added a comment.May 24 2019, 5:31 AM

Change 512311 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2062

https://gerrit.wikimedia.org/r/512311

Stashbot added a comment.May 24 2019, 5:34 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-24T05:34:25Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2062 from config T220170 (duration: 00m 49s)

Stashbot added a comment.May 24 2019, 5:35 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-24T05:35:38Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2062 from config T220170 (duration: 00m 48s)

gerritbot added a comment.May 24 2019, 5:42 AM

Change 512312 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db2062 from s1 to m1

https://gerrit.wikimedia.org/r/512312

gerritbot added a comment.May 24 2019, 5:46 AM

Change 512312 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2062 from s1 to m1

https://gerrit.wikimedia.org/r/512312

ops-monitoring-bot added a comment.May 24 2019, 5:50 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2062.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201905240550_marostegui_4423.log.

Stashbot added a comment.May 24 2019, 6:08 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-24T06:08:18Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to new hosts T220170 (duration: 00m 48s)

ops-monitoring-bot added a comment.May 24 2019, 6:16 AM

Completed auto-reimage of hosts:

['db2062.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.May 24 2019, 6:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-24T06:17:14Z] <marostegui> Stop MySQL on db2078:m1 to clone db2062 - T220170

Ladsgroup removed a project: Patch-For-Review.May 28 2019, 3:37 PM
• mobrovac added a parent task: T88445: MediaWiki active/active datacenter investigation and work (tracking).May 31 2019, 11:47 AM
Marostegui added a subtask: T224852: Failover s4 primary master: db1068 to db1081.Jun 3 2019, 5:50 AM
Stashbot added a comment.Jun 3 2019, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-03T13:19:03Z] <marostegui> Move db2078:3321 under db2062 T220170

Marostegui changed the status of subtask T211613: rack/setup/install db11[26-38].eqiad.wmnet from Open to Stalled.Jun 4 2019, 5:10 PM
gerritbot added a comment.Jun 5 2019, 5:21 AM

Change 514418 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2058db2065: Enable notifications

https://gerrit.wikimedia.org/r/514418

gerritbot added a project: Patch-For-Review.Jun 5 2019, 5:21 AM
gerritbot added a comment.Jun 5 2019, 5:30 AM

Change 514418 merged by Marostegui:
[operations/puppet@production] db2058,db2065: Enable notifications

https://gerrit.wikimedia.org/r/514418

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2019, 6:10 AM
Stashbot added a comment.Jun 5 2019, 6:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-05T06:17:58Z] <marostegui> Start topology changes on s4 codfw to replace current master db2051 with db2090 - T220170

gerritbot added a comment.Jun 5 2019, 6:21 AM

Change 514423 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Mimic codfw weights to eqiad

https://gerrit.wikimedia.org/r/514423

gerritbot added a project: Patch-For-Review.Jun 5 2019, 6:21 AM
gerritbot added a comment.Jun 5 2019, 6:24 AM

Change 514423 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Mimic codfw weights to eqiad

https://gerrit.wikimedia.org/r/514423

Stashbot added a comment.Jun 5 2019, 6:25 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-05T06:25:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Mimic s4 codfw weights to eqiad T220170 (duration: 00m 55s)

gerritbot added a comment.Jun 5 2019, 6:31 AM

Change 514424 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2090 to codfw s4 master

https://gerrit.wikimedia.org/r/514424

gerritbot added a comment.Jun 5 2019, 6:34 AM

Change 514424 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2090 to codfw s4 master

https://gerrit.wikimedia.org/r/514424

gerritbot added a comment.Jun 5 2019, 6:41 AM

Change 514425 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Promote db2090 to s4 codfw master

https://gerrit.wikimedia.org/r/514425

gerritbot added a comment.Jun 5 2019, 6:43 AM

Change 514426 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: db2051 disable notifications

https://gerrit.wikimedia.org/r/514426

gerritbot added a comment.Jun 5 2019, 6:44 AM

Change 514425 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Promote db2090 to s4 codfw master

https://gerrit.wikimedia.org/r/514425

gerritbot added a comment.Jun 5 2019, 6:44 AM

Change 514426 merged by Marostegui:
[operations/puppet@production] mariadb: db2051 disable notifications

https://gerrit.wikimedia.org/r/514426

Stashbot added a comment.Jun 5 2019, 6:45 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-05T06:45:23Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2090 to s4 codfw master T220170 (duration: 00m 54s)

Stashbot added a comment.Jun 5 2019, 6:45 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-05T06:45:29Z] <marostegui> Restart MySQL on db2110 to get the binlog format changed to STATEMENT - T220170

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2019, 7:10 AM
Stashbot added a comment.Jun 7 2019, 9:00 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-07T09:00:40Z] <marostegui> Upgrade x1 codfw hosts in preparation for its failover T220170

Marostegui closed subtask T224852: Failover s4 primary master: db1068 to db1081 as Resolved.Jun 19 2019, 5:23 AM
Marostegui changed the status of subtask T211613: rack/setup/install db11[26-38].eqiad.wmnet from Stalled to Open.Jun 25 2019, 7:38 AM
Marostegui closed subtask T211613: rack/setup/install db11[26-38].eqiad.wmnet as Resolved.
Marostegui added a subtask: T226358: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC.Jun 26 2019, 5:37 AM
Marostegui added a subscriber: mark.Jun 26 2019, 7:07 AM

I had a chat with @mark and we are considering this Q4 goal done:

I will not close this task, as it will be used for the remaining part of the goals related to this, which will happen in Q1

Marostegui updated the task description. (Show Details)Jun 27 2019, 8:32 AM
Marostegui added subtasks: T227062: Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required), T226952: Failover m2 master db1065 to db1132.Jul 3 2019, 8:47 AM
Marostegui closed subtask T226358: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC as Resolved.Jul 3 2019, 12:51 PM
Marostegui closed subtask T226952: Failover m2 master db1065 to db1132 as Resolved.Jul 9 2019, 7:30 AM
Marostegui mentioned this in Unknown Object (Task).Jul 17 2019, 7:11 AM
Marostegui added a subtask: Unknown Object (Task).Jul 22 2019, 12:46 PM
Papaul closed subtask Unknown Object (Task) as Resolved.Jul 22 2019, 4:06 PM
Marostegui added a subtask: T228243: Switchover m3 (phabricator) master db1072 to db1128.Jul 26 2019, 5:26 AM
Marostegui closed subtask T227062: Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required) as Resolved.Jul 30 2019, 5:48 AM
Marostegui added a subtask: T229657: Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC.Aug 2 2019, 12:31 PM
gerritbot added a comment.Aug 5 2019, 8:00 AM

Change 528070 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2107 to codfw s2 master

https://gerrit.wikimedia.org/r/528070

gerritbot added a project: Patch-For-Review.Aug 5 2019, 8:00 AM
Stashbot mentioned this in T221533: Decommission old coredb machines (<=db2042).Aug 5 2019, 8:21 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-05T08:21:27Z] <marostegui> Switchover s2 codfw master from db2035 to db2107 - T221533 T220170

gerritbot added a comment.Aug 5 2019, 8:28 AM

Change 528070 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2107 to codfw s2 master

https://gerrit.wikimedia.org/r/528070

Maintenance_bot removed a project: Patch-For-Review.Aug 5 2019, 9:11 AM
gerritbot added a comment.Aug 5 2019, 9:27 AM

Change 528081 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2105: Change binlog format

https://gerrit.wikimedia.org/r/528081

gerritbot added a project: Patch-For-Review.Aug 5 2019, 9:27 AM

Change 528081 merged by Marostegui:
[operations/puppet@production] db2105: Change binlog format

https://gerrit.wikimedia.org/r/528081

Maintenance_bot removed a project: Patch-For-Review.Aug 5 2019, 10:10 AM
Papaul closed subtask Unknown Object (Task) as Resolved.Aug 5 2019, 4:26 PM
Marostegui added a subtask: T221533: Decommission old coredb machines (<=db2042).Aug 7 2019, 9:51 AM
Marostegui closed subtask T221533: Decommission old coredb machines (<=db2042) as Resolved.Aug 8 2019, 5:57 AM
gerritbot added a comment.Aug 8 2019, 6:23 AM

Change 528987 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db2096 to x1 codfw master

https://gerrit.wikimedia.org/r/528987

gerritbot added a project: Patch-For-Review.Aug 8 2019, 6:23 AM

Change 528988 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Promote db2096 to x1 codfw master

https://gerrit.wikimedia.org/r/528988

Stashbot added a comment.Aug 8 2019, 7:39 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-08T07:39:13Z] <marostegui> Switchover x1 codfw master db2069 -> db2096 T220170

gerritbot added a comment.Aug 8 2019, 7:41 AM

Change 528988 merged by Marostegui:
[operations/mediawiki-config@master] db-codfw.php: Promote db2096 to x1 codfw master

https://gerrit.wikimedia.org/r/528988

gerritbot added a comment.Aug 8 2019, 7:45 AM

Change 528987 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db2096 to x1 codfw master

https://gerrit.wikimedia.org/r/528987

Stashbot added a comment.Aug 8 2019, 7:48 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-08T07:48:27Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2096 as codfw x1 master T220170 (duration: 00m 57s)

gerritbot added a comment.Aug 12 2019, 7:57 AM

Change 529707 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Reorganize s3 codfw weights

https://gerrit.wikimedia.org/r/529707

Stashbot added a comment.Aug 12 2019, 8:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-12T08:07:31Z] <marostegui@cumin1001> dbctl commit (dc=codfw): 'Reorganize s3 codfw weights T220170', diff saved to https://phabricator.wikimedia.org/P8901 and previous config saved to /var/cache/conftool/dbconfig/20190812-080731-marostegui.json

gerritbot added a comment.Aug 12 2019, 8:10 AM

Change 529707 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Reorganize s3 codfw weights

https://gerrit.wikimedia.org/r/529707

Stashbot added a comment.Aug 12 2019, 8:12 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-12T08:12:32Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Reorganize s3 codfw weights T220170 (duration: 00m 48s)

Maintenance_bot removed a project: Patch-For-Review.Aug 12 2019, 9:10 AM
Marostegui closed subtask T230459: Replace db2044 with db2063 as Declined.Aug 19 2019, 5:28 AM
Marostegui closed subtask T229657: Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC as Resolved.Sep 3 2019, 2:45 PM
Marostegui closed subtask T230106: Switchover codfw primary database masters to new hosts as Resolved.Sep 11 2019, 9:40 AM
Marostegui added a subtask: Unknown Object (Task).Oct 4 2019, 1:08 PM
jcrespo added a subtask: T237702: (codfw):rack/setup/install db213[2-5].Nov 8 2019, 2:23 PM
Papaul closed subtask Unknown Object (Task) as Resolved.Nov 8 2019, 7:15 PM
Marostegui closed subtask T237702: (codfw):rack/setup/install db213[2-5] as Resolved.Nov 13 2019, 6:53 AM
Marostegui closed this task as Resolved.Dec 12 2019, 6:02 AM

Everything is now done

Marostegui renamed this task from Address Database infrastructure blockers on datacenter switchover & multi-dc deployment to Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment.Dec 12 2019, 6:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits