Page MenuHomePhabricator
Create Task
Maniphest T124670

Figure out and document the datacenter switchover process
Closed, ResolvedPublic
Actions

Description

We need to figure out the process for switching from eqiad to codfw and back in our current hot/cold setup. We previously did something similar for the pmtpa->eqiad switchover, but this is wildly out of date by now. We'll need a new checklist and one we should keep up-to-date going forward.

Details

SubjectRepoBranchLines +/-
parsoid::testing: use master_dc variablesoperations/puppetproduction+17 -17
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 InvalidNoneT125673 Switch over from Eqiad to Codfw as primary datacentre
 Resolved• jcrespoT124670 Figure out and document the datacenter switchover process
 ResolvedKrinkleT124671 Switchover of the application servers to codfw
 ResolvedNoneT124879 Check the memcached and redis (sessions) configuration and functionality in codfw
 Resolved• jcrespoT124697 Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming
 Resolved• jcrespoT125386 Clarify how mysql dumps will be architectured during codfw failover
 Resolved• jcrespoT124795 codfw is in read only according to mediawiki
 ResolvedJoeT126242 Reduce the number of appservers we're using in eqiad
 Resolved• CmjohnsonT129060 mw1026-69 are shut down and should be physically decommissioned
 Resolved• CmjohnsonT133770 mw1070-89 and mw1121-30 are shut down and should be physically decommissioned
 ResolvedJoeT129062 Reorganize mw2* servers across clusters
 ResolvedJoeT124672 Check the redis (jobqueue) configuration in codfw
 ResolvedRobHT126453 codfw: (2) servers for redis jobrunners
 ResolvedelukeyT129178 rack/setup/deploy rdb200[5-6]
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
 ResolvedJoeT126470 Enable Redis cross-dc replication
 ResolvedKrinkleT124673 Figure out how to migrate the jobqueues
 ResolvedaaronT129250 Ensure post-send handlers check and respect read-only-mode
 ResolvedKrinkleT129258 Ensure maintainers of long-running scripts on terbium expect downtime for switchover
 ResolvedelukeyT123675 Reinstall redis servers (Job queues) with Jessie (NOTE: rdb1002 is special and is excluded!)
 ResolvedelukeyT128730 Sudden increase in NOTICE events from hhvm while trying to de-pool rdb1003 for maintenance
 ResolvedJoeT129317 Dedicate 1/2 codfw jobrunners to gwtoolset jobs
 ResolvedJoeT126235 Have a strategy to switch restbase to use services in the appropriate datacenter
 Resolved• jcrespoT129236 Improve documentation about database switchover

Event Timeline

faidon created this task.Jan 25 2016, 4:42 PM
faidon raised the priority of this task from to Medium.
faidon updated the task description. (Show Details)
faidon added projects: SRE, codfw-rollout, codfw-rollout-Jan-Mar-2016.
faidon subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2016, 4:42 PM
Krinkle added a parent task: T125673: Switch over from Eqiad to Codfw as primary datacentre.Feb 3 2016, 4:42 PM
Krinkle subscribed.

Might be a duplicate of T114398: Make and document DC switch-over scripts.

gerritbot subscribed.Mar 8 2016, 2:18 PM

Change 275814 had a related patch set uploaded (by Giuseppe Lavagetto):
parsoid::testing: use master_dc variables

https://gerrit.wikimedia.org/r/275814

gerritbot added a project: Patch-For-Review.Mar 8 2016, 2:18 PM
• jcrespo created subtask T129236: Improve documentation about database switchover.Mar 8 2016, 4:53 PM
Krinkle unsubscribed.Mar 8 2016, 9:44 PM
Joe closed subtask T126235: Have a strategy to switch restbase to use services in the appropriate datacenter as Resolved.Mar 16 2016, 7:48 AM
gerritbot added a comment.Mar 16 2016, 9:31 AM

Change 275814 merged by Giuseppe Lavagetto:
parsoid::testing: use master_dc variables

https://gerrit.wikimedia.org/r/275814

• jcrespo subscribed.Mar 16 2016, 12:19 PM

This is a duplicate, but I would merge T114398 into it, as this one has activity.

Joe closed subtask T124672: Check the redis (jobqueue) configuration in codfw as Resolved.Mar 23 2016, 8:22 AM
Joe moved this task from Backlog to In Progress on the codfw-rollout-Jan-Mar-2016 board.Mar 31 2016, 11:37 AM
Krinkle closed subtask T124671: Switchover of the application servers to codfw as Resolved.Apr 21 2016, 3:07 PM
Krinkle closed subtask T124673: Figure out how to migrate the jobqueues as Resolved.Apr 21 2016, 3:18 PM
Krinkle added a subscriber: aaron.Apr 21 2016, 3:21 PM
In T114398, @aaron wrote:>

See also T114271.

We need scripts and processes to do a planned switch from master datacenter A to B:

a) Go read-only on the app level (mostly MediaWiki)
b) Make sure write traffic stops
c) Go read-only for all data stores
d) Wait for all data stores in the B datacenter to catch up and be in sync with A
e) Make the B datacenter the new master datacenter (systems and app level)
f) End read-only mode

Read-only mode should be as short as possible so we can actually test this.

Krinkle merged a task: T114398: Make and document DC switch-over scripts.Apr 21 2016, 3:21 PM
Krinkle removed a subtask: T125069: Create a service location / discovery system for locating local/master resources easily across all WMF applications.
Krinkle added subscribers: • bd808, PleaseStand, Krenair and 9 others.
• jcrespo closed subtask T129236: Improve documentation about database switchover as Resolved.Apr 28 2016, 9:49 AM

"we should keep up-to-date going forward" is not really a finite actionable task, I would consider this Resolved. There are multiple things to fix- but on the process itself (e.g. better database orchestration), not only on the documentation.

• jcrespo added a comment.EditedApr 28 2016, 9:53 AM

Process documented on: https://wikitech.wikimedia.org/wiki/Switch_Datacenter

• jcrespo closed this task as Resolved.Apr 28 2016, 9:53 AM
• jcrespo claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits