Page MenuHomePhabricator

June 2021 Datacenter switchover
Closed, ResolvedPublic

Description

This is the meta task for the June 2021 Datacenter switchover (eqiad -> codfw).

Schedule:

Services: Monday, June 28th, 2021 14:00 UTC
Traffic: Monday, June 28th, 2021 15:00 UTC
MediaWiki: Tuesday, June 29th, 2021 14:00 UTC

Switching back: TBD, but at least 1 month later

See also: https://wikitech.wikimedia.org/wiki/Switch_Datacenter - section Schedule

Related Objects

StatusSubtypeAssignedTask
ResolvedMarostegui
DeclinedNone
ResolvedMarostegui
ResolvedJclark-ctr
ResolvedMarostegui
ResolvedMarostegui
ResolvedRequestwiki_willy
ResolvedLegoktm
Resolvedsgrabarczuk
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedAndrew
ResolvedMarostegui
ResolvedAndrew
DeclinedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedLadsgroup
DuplicateNone
ResolvedBstorm
ResolvedMarostegui
ResolvedBTullis
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedKormat
ResolvedMarostegui
ResolvedTrizek-WMF
ResolvedKormat
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
Resolvedsgrabarczuk
ResolvedMarostegui
Resolvedsgrabarczuk
ResolvedCmjohnson
ResolvedMarostegui
ResolvedMarostegui
Resolvedsgrabarczuk
ResolvedRequestCmjohnson
ResolvedMarostegui
ResolvedRequestwiki_willy
ResolvedRequestCmjohnson
ResolvedRequestCmjohnson
ResolvedRequestCmjohnson
ResolvedRequestCmjohnson
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedKormat
ResolvedKormat
ResolvedTrizek-WMF
ResolvedMarostegui
ResolvedMarostegui
Resolvedsgrabarczuk
ResolvedMarostegui
ResolvedKormat
ResolvedMarostegui
ResolvedMarostegui
ResolvedKormat
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui

Event Timeline

Dzahn triaged this task as Medium priority.May 3 2021, 7:04 PM
Aklapper added a parent task: Restricted Task.May 17 2021, 9:43 AM

Change 701610 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] admin_state: depool eqiad for datacenter switchover (June 2021)

https://gerrit.wikimedia.org/r/701610

Change 701610 merged by Ssingh:

[operations/dns@master] admin_state: depool eqiad for datacenter switchover (June 2021)

https://gerrit.wikimedia.org/r/701610

Mentioned in SAL (#wikimedia-operations) [2021-06-28T18:40:34Z] <ebernhardson@deploy1002> Synchronized wmf-config/: T281515: Prepare Cirrus more_like for dc switchover (duration: 01m 02s)

I did a successful run through of the live-test mode just now, where we "switch" from codfw -> eqiad. The only issue I ran into is T285519#7182377, which I live-hacked a fix for.

Change 702128 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Change masters cnames

https://gerrit.wikimedia.org/r/702128

Change 702128 merged by Marostegui:

[operations/dns@master] wmnet: Change masters cnames

https://gerrit.wikimedia.org/r/702128

The switchover is mostly complete now, we were read only from 2021-06-29 14:21:26.671853 to 2021-06-29 14:23:23.504447, or 1m57s.

The raw notes of the issues we encountered are at https://etherpad.wikimedia.org/p/2021-switchdc-notes, later today I'll distill those into actionable Phabricator tasks and write up a report for how it went and what should be improved.

Legoktm claimed this task.

A recap blog post was published a few days ago: https://techblog.wikimedia.org/2021/07/23/june-2021-data-center-switchover/

T287539: September 2021 Datacenter switchover (codfw -> eqiad) tracks switching back to eqiad in September, closing this as resolved accordingly.