Page MenuHomePhabricator
Create Task
Maniphest T281515

June 2021 Datacenter switchover
Closed, ResolvedPublic
Actions

Description

This is the meta task for the June 2021 Datacenter switchover (eqiad -> codfw).

Schedule:

Services: Monday, June 28th, 2021 14:00 UTC
Traffic: Monday, June 28th, 2021 15:00 UTC
MediaWiki: Tuesday, June 29th, 2021 14:00 UTC

Switching back: TBD, but at least 1 month later

See also: https://wikitech.wikimedia.org/wiki/Switch_Datacenter - section Schedule

Details

SubjectRepoBranchLines +/-
admin_state: depool eqiad for datacenter switchover (June 2021)operations/dnsmaster+1 -0
wmnet: Change masters cnamesoperations/dnsmaster+18 -18
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedMarosteguiT258361 Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers)
 DeclinedNoneT258386 db1080-95 batch possibly suffering BBU issues
 ResolvedMarosteguiT258336 db1082 crashed
 ResolvedJclark-ctrT258910 Remove db1082's BBU on-site
 ResolvedMarosteguiT258360 db1085 crashed
 ResolvedMarosteguiT264755 db1076 crashed - BBU failure
 ResolvedRequestwiki_willyT267088 decommission db1091.eqiad.wmnet
Restricted Task
 ResolvedLegoktmT281515 June 2021 Datacenter switchover
 ResolvedsgrabarczukT281209 CommRel support for June 2021 Switchover
 ResolvedMarosteguiT284897 Pre DC switchover eqiad -> codfw DB work
 ResolvedMarosteguiT285139 DB maintenance work to do while eqiad is passive (June 2021)
 ResolvedMarosteguiT167973 Move database for wikitech (labswiki) to a main cluster section
 ResolvedAndrewT188029 Move labswiki database to m5
ResolvedMarosteguiT282074 Audit labswiki grants
 ResolvedAndrewT282209 Enable connection from labweb1001/labweb1002 to s6
 DeclinedAndrewT282573 Move labweb/cloudweb hosts to private IPs
 ResolvedAndrewT284106 Replicate & sanitize wikitech data
 ResolvedAndrewT284108 Bring labswiki database tables up to date
 ResolvedLadsgroupT269348 wikitech database has almost all of its varbinary fields wrong
 DuplicateNoneT284736 Create views on labswiki on clouddb* hosts
 Resolved• BstormT287442 Create views for labswiki (wikitech)
 ResolvedMarosteguiT283093 Schema change for making cuc_id in cu_changes unsigned
Restricted Task
Restricted Task
Restricted Task
 ResolvedBTullisT298940 Reimage WMCS db proxies to Bullseye
 ResolvedMarosteguiT287852 Failover m2 master (db1107) to a different host to upgrade its kernel
 ResolvedMarosteguiT288197 Failover m3 (phabricator) master (db1132) to a different host to upgrade its kernel
 ResolvedMarosteguiT266483 Enable report_host for mariadb
 Resolved• KormatT271106 Enable report_host on candidate masters
 ResolvedMarosteguiT271427 Switchover s4 (commonswiki) from db1081 to db1138
 ResolvedTrizek-WMFT271791 Database primary master failover on s4 (commonswiki)
 Resolved• KormatT272954 Fix db-switchover update zarcillo part
 ResolvedMarosteguiT272568 Add m* and es4/es5 sections to Orchestrator
 ResolvedMarosteguiT271540 Upgrade and restart m1 master (db1080)
 ResolvedMarosteguiT272596 Restart m3 (phabricator) database master db1132
 ResolvedMarosteguiT272388 Restart m5 master (db1128)
 ResolvedMarosteguiT272964 Restart m2 database master (db1107)
 ResolvedMarosteguiT273758 Restart x1 database master (db1103)
 ResolvedsgrabarczukT273762 Read only time for extension 1 (x1) primary database on 2021-02-17
 ResolvedMarosteguiT274336 Switchover s7 from db1086 to db1136
 ResolvedsgrabarczukT276899 Read only time window needed for s7
 Resolved• CmjohnsonT277007 Upgrade firmware on db1136
 ResolvedMarosteguiT278214 Switchover s1 from db1083 to db1163
 ResolvedMarosteguiT276150 Schema change to make rc_id unsigned and rc_timestamp BINARY
 ResolvedsgrabarczukT279505 Read only time window needed for s1 (enwiki)
 ResolvedRequest• CmjohnsonT281445 decommission db1083.eqiad.wmnet
 ResolvedMarosteguiT280492 Upgrade all sanitarium masters to 10.4 and Buster
 ResolvedRequestwiki_willyT281794 decommission db1082.eqiad.wmnet
 ResolvedRequest• CmjohnsonT281959 decommission db1074.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282079 decommission db1079.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282093 decommission db1087.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282096 decommission db1085.eqiad.wmnet
 ResolvedMarosteguiT284375 Rename name_title index on page to page_name_title
 ResolvedMarosteguiT284619 Schema change for renaming several indexes in change_tag table
 ResolvedMarosteguiT273360 Schema change for dropping default of img_timestamp and making it binary(14)
 ResolvedMarosteguiT268392 Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14)
 ResolvedMarosteguiT266486 Schema change to turn user_last_timestamp.user_newtalk to binary(14)
 Resolved• KormatT283131 Upgrade s3 to Debian Buster and MariaDB 10.4
 Resolved• KormatT284648 Switchover s3 from db1123 to db1157
 ResolvedTrizek-WMFT284650 Read-only window needed for s3 (~900 wikis)
 ResolvedMarosteguiT283235 Upgrade s5 to Debian Buster and MariaDB 10.4
 ResolvedMarosteguiT284529 Switchover s5 from db1100 to db1130
 ResolvedsgrabarczukT284530 Read only time window needed for s5
 ResolvedMarosteguiT277123 Extend iwlinks.iwl_prefix to VARBINARY(32)
 Resolved• KormatT277118 iw_url in interwiki is varbinary(127) in production but blob in code
 ResolvedMarosteguiT283499 Schema change for renaming page_timestamp index on revision table to rev_page_timestamp
 ResolvedMarosteguiT278621 job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007
 Resolved• KormatT277116 fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production
 ResolvedMarosteguiT287344 Please optimize logging table in dewiki
 ResolvedMarosteguiT288273 Please optimize image table in commonswiki
 ResolvedMarosteguiT290057 Optimize flaggedtemplates tables in production
 ResolvedMarosteguiT286102 Please optimize logging table in ruwiki
 ResolvedMarosteguiT270620 Schema change for renaming several indexes in logging table

Event Timeline

Legoktm created this task.Apr 29 2021, 7:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 29 2021, 7:04 PM
Legoktm added a subtask: T281209: CommRel support for June 2021 Switchover.Apr 29 2021, 7:04 PM
taavi subscribed.Apr 29 2021, 7:04 PM
Krinkle added a project: Performance-Team (Radar).Apr 29 2021, 8:18 PM
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.
Marostegui subscribed.Apr 29 2021, 8:19 PM
LSobanski subscribed.Apr 29 2021, 9:04 PM
Peachey88 subscribed.Apr 29 2021, 9:14 PM
Volans subscribed.Apr 29 2021, 10:10 PM
Marostegui mentioned this in T167973: Move database for wikitech (labswiki) to a main cluster section.Apr 30 2021, 12:47 PM
EBernhardson subscribed.May 3 2021, 3:04 PM
Dzahn triaged this task as Medium priority.May 3 2021, 7:04 PM
Aklapper added a parent task: Restricted Task.May 17 2021, 9:43 AM
• wkandek subscribed.May 19 2021, 4:16 PM
• Kormat subscribed.Jun 9 2021, 12:19 PM
Kaartic subscribed.Jun 15 2021, 8:53 AM
Legoktm added a subtask: T284897: Pre DC switchover eqiad -> codfw DB work.Jun 17 2021, 4:47 PM
Legoktm removed a subtask: T285069: Check "Days in advance preparation" for databases before DC switchover.
Ladsgroup subscribed.Jun 18 2021, 3:06 AM
Marostegui added a subtask: T285139: DB maintenance work to do while eqiad is passive (June 2021).Jun 18 2021, 11:21 AM
• wkandek added a project: serviceops.Jun 21 2021, 4:39 AM
• wkandek moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.
lmata subscribed.Jun 21 2021, 5:28 AM
Marostegui closed subtask T284897: Pre DC switchover eqiad -> codfw DB work as Resolved.Jun 24 2021, 9:24 AM
JMeybohm subscribed.Jun 24 2021, 11:27 AM
• wkandek updated the task description. (Show Details)Jun 24 2021, 5:46 PM
ssingh subscribed.Jun 25 2021, 8:23 PM
gerritbot added a comment.Jun 25 2021, 8:32 PM

Change 701610 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] admin_state: depool eqiad for datacenter switchover (June 2021)

https://gerrit.wikimedia.org/r/701610

gerritbot added a project: Patch-For-Review.Jun 25 2021, 8:32 PM
ReaperDawn subscribed.Jun 28 2021, 4:37 AM
gerritbot added a comment.Jun 28 2021, 3:43 PM

Change 701610 merged by Ssingh:

[operations/dns@master] admin_state: depool eqiad for datacenter switchover (June 2021)

https://gerrit.wikimedia.org/r/701610

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2021, 4:10 PM
Stashbot added a comment.Jun 28 2021, 6:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-28T18:40:34Z] <ebernhardson@deploy1002> Synchronized wmf-config/: T281515: Prepare Cirrus more_like for dc switchover (duration: 01m 02s)

Legoktm added a comment.Jun 28 2021, 7:35 PM

I posted an update on today's switchover to wikitech-l: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/XI57Z6T2DK7IC345VFTENM5RLTQBQDEQ/

Legoktm added a comment.Jun 28 2021, 10:56 PM

I did a successful run through of the live-test mode just now, where we "switch" from codfw -> eqiad. The only issue I ran into is T285519#7182377, which I live-hacked a fix for.

R4356th subscribed.Jun 29 2021, 5:32 AM
gerritbot added a comment.Jun 29 2021, 1:38 PM

Change 702128 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Change masters cnames

https://gerrit.wikimedia.org/r/702128

gerritbot added a project: Patch-For-Review.Jun 29 2021, 1:38 PM
Agusbou2015 subscribed.Jun 29 2021, 1:44 PM
gerritbot added a comment.Jun 29 2021, 2:47 PM

Change 702128 merged by Marostegui:

[operations/dns@master] wmnet: Change masters cnames

https://gerrit.wikimedia.org/r/702128

Maintenance_bot removed a project: Patch-For-Review.Jun 29 2021, 3:10 PM
Legoktm added a comment.Jun 29 2021, 3:42 PM

The switchover is mostly complete now, we were read only from 2021-06-29 14:21:26.671853 to 2021-06-29 14:23:23.504447, or 1m57s.

The raw notes of the issues we encountered are at https://etherpad.wikimedia.org/p/2021-switchdc-notes, later today I'll distill those into actionable Phabricator tasks and write up a report for how it went and what should be improved.

sgrabarczuk closed subtask T281209: CommRel support for June 2021 Switchover as Resolved.Jun 29 2021, 4:44 PM
Zabe subscribed.Jun 29 2021, 6:05 PM
Legoktm added a comment.Jun 30 2021, 1:52 AM

Bugs filed as a result of today's switchover:

Legoktm added a comment.Jun 30 2021, 2:20 AM

Today's summary: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/ENL3P5SA7RSOHPN4ILMXQ2BGBF5XR776/

Legoktm mentioned this in T287539: September 2021 Datacenter switchover (codfw -> eqiad).Jul 27 2021, 11:45 PM
Legoktm closed this task as Resolved.Jul 28 2021, 12:59 AM
Legoktm claimed this task.

A recap blog post was published a few days ago: https://techblog.wikimedia.org/2021/07/23/june-2021-data-center-switchover/

T287539: September 2021 Datacenter switchover (codfw -> eqiad) tracks switching back to eqiad in September, closing this as resolved accordingly.

Marostegui closed subtask T285139: DB maintenance work to do while eqiad is passive (June 2021) as Resolved.Oct 20 2021, 3:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL