Page MenuHomePhabricator
Create Task
Maniphest T243316

FY2020-2021 Q1 eqiad -> codfw switchover
Closed, ResolvedPublic
Actions

Description

Sub-umbrella task for work blocking the switchover from eqiad to codfw.

Details

SubjectRepoBranchLines +/-
Depool eqiad for 2020-2021 DC switchoveroperations/dnsmaster+1 -0
sre.switchdc.mediawiki: adapt to current statusoperations/cookbooksmaster+23 -83
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRLazarusT243314 FY2020-2021 Q1 DC switchover and switchback
 ResolvedRLazarusT243316 FY2020-2021 Q1 eqiad -> codfw switchover
 Resolved• MarosteguiT243373 Enable DB replication codfw -> eqiad before the switchover and some other checks
 Resolved• KormatT243935 Audit all cumin queries in switchdc scripts
 Resolved• MarosteguiT260042 Compare a few tables per section before the switchover
 ResolvedDzahnT261159 assess and re-evaluate 'weight' settings of appservers in codfw
 Resolved• MarosteguiT186188 Failover DB masters in row D
 Resolved• MarosteguiT224852 Failover s4 primary master: db1068 to db1081
 ResolvedJohanT224516 Database primary master failover on s4 (commonswiki)
 Resolved• MarosteguiT227062 Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required)
 ResolvedJohanT227063 Database primary master failover on s8 (wikidatawiki)
 Resolved• MarosteguiT234300 Switchover s5 primary database master db1070 -> db1100 - 15th Oct 05:00 - 05:30 UTC
 ResolvedTrizek-WMFT234303 Community Relations support needed a several read-only window for s5
 Resolved• MarosteguiT238044 Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC
 ResolvedTrizek-WMFT238046 Community Relations support needed for a read-only window for s7 (several wikis)
 Resolved• KormatT239238 Switchover s8 primary database master db1109 -> db1104 - 2020-09-29 08:00 UTC
 Resolved• CmjohnsonT262901 Physically move db1131 from B5 to C8
 Resolved• MarosteguiT263227 Failover s6 master, db1093 to db1131
 ResolvedDzahnT260654 Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420)
 ResolvedPapaulT261524 decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part

Event Timeline

RLazarus created this task.Jan 21 2020, 6:42 PM
• Marostegui triaged this task as Medium priority.Jan 22 2020, 6:11 AM
gerritbot added a comment.Feb 4 2020, 8:02 PM

Change 570131 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.switchdc.mediawiki: adapt to current status

https://gerrit.wikimedia.org/r/570131

gerritbot added a project: Patch-For-Review.Feb 4 2020, 8:02 PM
RLazarus renamed this task from 2020 Q3 eqiad -> codfw switchover to 2020 Q4 eqiad -> codfw switchover.Feb 10 2020, 10:43 PM
Aklapper renamed this task from 2020 Q4 eqiad -> codfw switchover to FY2019-20 Q4 eqiad -> codfw switchover.Feb 11 2020, 2:10 AM
gerritbot added a comment.Feb 12 2020, 7:54 AM

Change 570131 merged by jenkins-bot:
[operations/cookbooks@master] sre.switchdc.mediawiki: adapt to current status

https://gerrit.wikimedia.org/r/570131

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2020, 8:10 AM
RLazarus renamed this task from FY2019-20 Q4 eqiad -> codfw switchover to FY2020-2021 Q1 eqiad -> codfw switchover.May 4 2020, 7:04 PM
RLazarus mentioned this in T252391: Reimage one memcached shard per DC to Buster.Jun 23 2020, 4:33 PM
• Kormat closed subtask T243935: Audit all cumin queries in switchdc scripts as Resolved.Jul 1 2020, 10:57 AM
RhinosF1 subscribed.Aug 3 2020, 2:23 PM
Dzahn closed subtask T261159: assess and re-evaluate 'weight' settings of appservers in codfw as Resolved.Aug 27 2020, 6:33 PM
• Marostegui closed subtask T243373: Enable DB replication codfw -> eqiad before the switchover and some other checks as Resolved.Aug 28 2020, 8:43 AM
• Marostegui added a subtask: T186188: Failover DB masters in row D.Aug 28 2020, 10:35 AM
• Marostegui changed the status of subtask T186188: Failover DB masters in row D from Stalled to Open.
Dzahn added a subtask: T260654: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420).Aug 28 2020, 5:43 PM
Dzahn closed subtask T260654: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) as Resolved.Aug 28 2020, 7:21 PM
• Marostegui closed subtask T260042: Compare a few tables per section before the switchover as Resolved.Aug 31 2020, 12:55 PM
gerritbot added a comment.Aug 31 2020, 2:56 PM

Change 623360 had a related patch set uploaded (by Ema; owner: Ema):
[operations/dns@master] Depool eqiad for 2020-2021 DC switchover

https://gerrit.wikimedia.org/r/623360

gerritbot added a project: Patch-For-Review.Aug 31 2020, 2:56 PM

Change 623360 merged by Ema:
[operations/dns@master] Depool eqiad for 2020-2021 DC switchover

https://gerrit.wikimedia.org/r/623360

Stashbot added a comment.Aug 31 2020, 2:58 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-31T14:58:26Z] <ema> Traffic: depool eqiad from user traffic T243316

Maintenance_bot removed a project: Patch-For-Review.Aug 31 2020, 3:10 PM
• Marostegui closed subtask T186188: Failover DB masters in row D as Resolved.Sep 30 2020, 7:51 AM
RLazarus closed this task as Resolved.Jan 11 2021, 3:55 PM
Volans mentioned this in rCCKBdc61c76ec3ab: sre.switchdc.mediawiki: adapt to current status.Dec 14 2022, 3:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits