Page MenuHomePhabricator
Create Task
Maniphest T243314

FY2020-2021 Q1 DC switchover and switchback
Closed, ResolvedPublic
Actions

Description

Umbrella task for all work related to the eqiad->codfw switchover planned for 2020-21 Q1, and the subsequent codfw->eqiad switchback.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRLazarusT243314 FY2020-2021 Q1 DC switchover and switchback
 ResolvedRLazarusT243316 FY2020-2021 Q1 eqiad -> codfw switchover
 Resolved• MarosteguiT243373 Enable DB replication codfw -> eqiad before the switchover and some other checks
 ResolvedKormatT243935 Audit all cumin queries in switchdc scripts
 Resolved• MarosteguiT260042 Compare a few tables per section before the switchover
 ResolvedDzahnT261159 assess and re-evaluate 'weight' settings of appservers in codfw
 Resolved• MarosteguiT186188 Failover DB masters in row D
 Resolved• MarosteguiT224852 Failover s4 primary master: db1068 to db1081
 ResolvedJohanT224516 Database primary master failover on s4 (commonswiki)
 Resolved• MarosteguiT227062 Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required)
 ResolvedJohanT227063 Database primary master failover on s8 (wikidatawiki)
 Resolved• MarosteguiT234300 Switchover s5 primary database master db1070 -> db1100 - 15th Oct 05:00 - 05:30 UTC
 ResolvedTrizek-WMFT234303 Community Relations support needed a several read-only window for s5
 Resolved• MarosteguiT238044 Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC
 ResolvedTrizek-WMFT238046 Community Relations support needed for a read-only window for s7 (several wikis)
 Resolved• CmjohnsonT262901 Physically move db1131 from B5 to C8
 Resolved• MarosteguiT263227 Failover s6 master, db1093 to db1131
 ResolvedDzahnT260654 Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420)
 ResolvedPapaulT261524 decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part
 ResolvedRLazarusT243318 FY2020-2021 Q1 codfw -> eqiad switchback
 Resolved• MarosteguiT237120 Schema change on production for increase the size of wbt_text_in_lang.wbxl_language
 ResolvedKormatT239238 Switchover s8 primary database master db1109 -> db1104 - 2020-09-29 08:00 UTC
 DuplicateNoneT243374 Enable DB replication eqiad -> codfw before the switchover
 Resolved• MarosteguiT232446 Compress new Wikibase tables
 DeclinedNoneT245107 Possibly replace db1087 (s8) with db1127 (x1) due to disk space constrains
 Resolved• MarosteguiT250055 Remove image.img_deleted column from production
 Resolved• MarosteguiT253276 Normalise MW Core database language fields length
 Resolved• MarosteguiT254462 Compress enwiki InnoDB tables
 Resolved• MarosteguiT238966 Apply updates for MCR, actor migration, and content migration, to production wikis.
 Resolved• BstormT252219 Drop MCR-obsoleted fields from the wiki replicas
 ResolveddanielT253289 Remove USE INDEX usertext_timestamp and other references from code
 ResolveddanielT258480 Query plan changes on enwiki.revision queries with MCR change
 Resolved• MarosteguiT259524 Review revision table and make sure that the PK is always rev_id
 Resolved• MarosteguiT256685 pl_namespace index on pagelinks is unique only in s8
 ResolvedKormatT259831 Schema change to make change_tag.ct_rc_id unsigned
 Resolved• MarosteguiT261914 Enable replication eqiad -> codfw and other checks
 ResolvedTrizek-WMFT264364 CommRel support for FY2020-2021 Q2 DC switchback
 Resolved• MarosteguiT260111 All sorts of random drifts in wikis in s3
 ResolvedRLazarusT211250 Create a mediawiki::cronjob define
 OpenNoneT250231 purgeParserCache.php: Cannot purge this kind of parser cache
 ResolvedLadsgroupT288175 Migrate wikibase-dispatch-changes crons to systemd timers
 DuplicateRLazarusT243520 Decommission the "session redis" cluster
 ResolvedTrizek-WMFT244808 CommRel support for FY2020-2021 Q1 DC switchover
 OpenNoneT207385 Create a check on the DC failover script to see if codfw -> eqiad replication is working before failing over to codfw (considering eqiad as the active DC by default)
 Resolved• emaT261312 Verify ATS handling of DNS TTLs

Event Timeline

RLazarus created this task.Jan 21 2020, 6:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 21 2020, 6:40 PM
• Marostegui triaged this task as Medium priority.Jan 22 2020, 6:11 AM
• Marostegui added a project: Goal.Jan 23 2020, 9:31 AM
Joe added a subtask: T211250: Create a mediawiki::cronjob define.Jan 23 2020, 2:43 PM
Ladsgroup subscribed.Feb 10 2020, 3:48 PM
RLazarus renamed this task from 2020 Q3 DC switchover and switchback to 2020 Q4 DC switchover and switchback.Feb 10 2020, 10:43 PM
RLazarus updated the task description. (Show Details)
Aklapper renamed this task from 2020 Q4 DC switchover and switchback to FY2019-20 Q4 DC switchover and switchback.Feb 11 2020, 2:10 AM
jcrespo added a subtask: T207385: Create a check on the DC failover script to see if codfw -> eqiad replication is working before failing over to codfw (considering eqiad as the active DC by default).Apr 29 2020, 6:54 AM
RLazarus renamed this task from FY2019-20 Q4 DC switchover and switchback to FY2020-2021 Q1 DC switchover and switchback.May 4 2020, 7:04 PM
RLazarus updated the task description. (Show Details)
• Elitre changed the status of subtask T244808: CommRel support for FY2020-2021 Q1 DC switchover from Open to Stalled.May 11 2020, 10:36 AM
Krinkle mentioned this in T254634: Determine and implement multi-dc strategy for ChronologyProtector.Jun 6 2020, 3:42 AM
Trizek-WMF changed the status of subtask T244808: CommRel support for FY2020-2021 Q1 DC switchover from Stalled to Open.Aug 3 2020, 2:19 PM
RhinosF1 subscribed.Aug 3 2020, 2:22 PM
bd808 subscribed.Aug 10 2020, 10:11 PM
taavi subscribed.Aug 11 2020, 11:00 AM
Agusbou2015 subscribed.Aug 24 2020, 11:52 PM
• ema closed subtask T261312: Verify ATS handling of DNS TTLs as Resolved.Aug 28 2020, 8:15 AM
gerritbot added a comment.Aug 31 2020, 1:14 PM

Change 623360 had a related patch set uploaded (by Ema; owner: Ema):
[operations/dns@master] Depool eqiad for 2020-2021 DC switchover

https://gerrit.wikimedia.org/r/623360

gerritbot added a project: Patch-For-Review.Aug 31 2020, 1:14 PM
Trizek-WMF closed subtask T244808: CommRel support for FY2020-2021 Q1 DC switchover as Resolved.Sep 14 2020, 2:24 PM
• Marostegui added a comment.Jan 8 2021, 12:40 PM

@RLazarus can this be closed?

RLazarus closed this task as Resolved.Jan 11 2021, 3:54 PM

Yep!

RLazarus closed subtask T243316: FY2020-2021 Q1 eqiad -> codfw switchover as Resolved.Jan 11 2021, 3:55 PM
RLazarus closed subtask T243318: FY2020-2021 Q1 codfw -> eqiad switchback as Resolved.
RLazarus closed subtask T211250: Create a mediawiki::cronjob define as Resolved.Aug 17 2021, 7:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL