Page MenuHomePhabricator
Create Task
Maniphest T125673

Switch over from Eqiad to Codfw as primary datacentre
Closed, InvalidPublic
Actions

Description

See also T88445: MediaWiki active/active datacenter investigation and work (tracking)

Details

SubjectRepoBranchLines +/-
iegreview: use $::parsoid_siteoperations/puppetproduction+2 -2
realm: add $::app_routes hashoperations/puppetproduction+19 -4
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 InvalidNoneT125673 Switch over from Eqiad to Codfw as primary datacentre
 DuplicateNoneT114398 Make and document DC switch-over scripts
 ResolvedBBlackT125510 Traffic Infrastructure support for Mar 2016 codfw rollout
 DuplicateNoneT127483 VCL source-DC switching: add "direct" capability
 ResolvedBBlackT127484 Refactor VCL for applayer datacenter-switching
 ResolvedBBlackT127492 Switch ulsfo to backend to codfw rather than eqiad
 ResolvedBBlackT127481 Refactor VCL for 3-tier capabilities and source-DC switching
 Resolved• emaT129424 Varnish support for shutting users out of a DC
 OpenNoneT114271 Audit *all* datastores to see how DR failover is handled
 ResolvedaaronT122387 Log multi-DB transaction in MediaWiki
 ResolvedjcrespoT124670 Figure out and document the datacenter switchover process
 ResolvedKrinkleT124671 Switchover of the application servers to codfw
 ResolvedNoneT124879 Check the memcached and redis (sessions) configuration and functionality in codfw
 ResolvedjcrespoT124697 Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming
 ResolvedjcrespoT125386 Clarify how mysql dumps will be architectured during codfw failover
 ResolvedJoeT126242 Reduce the number of appservers we're using in eqiad
 Resolved• CmjohnsonT129060 mw1026-69 are shut down and should be physically decommissioned
 Resolved• CmjohnsonT133770 mw1070-89 and mw1121-30 are shut down and should be physically decommissioned
 ResolvedJoeT129062 Reorganize mw2* servers across clusters
 ResolvedJoeT124672 Check the redis (jobqueue) configuration in codfw
 ResolvedRobHT126453 codfw: (2) servers for redis jobrunners
 ResolvedelukeyT129178 rack/setup/deploy rdb200[5-6]
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
 ResolvedJoeT126470 Enable Redis cross-dc replication
 ResolvedKrinkleT124673 Figure out how to migrate the jobqueues
 ResolvedaaronT129250 Ensure post-send handlers check and respect read-only-mode
 ResolvedKrinkleT129258 Ensure maintainers of long-running scripts on terbium expect downtime for switchover
 ResolvedelukeyT123675 Reinstall redis servers (Job queues) with Jessie (NOTE: rdb1002 is special and is excluded!)
 ResolvedelukeyT128730 Sudden increase in NOTICE events from hhvm while trying to de-pool rdb1003 for maintenance
 ResolvedJoeT129317 Dedicate 1/2 codfw jobrunners to gwtoolset jobs
 ResolvedJoeT126235 Have a strategy to switch restbase to use services in the appropriate datacenter
 ResolvedjcrespoT129236 Improve documentation about database switchover
 ResolvedVolansT124699 Change configuration to make codfw db masters as the masters of all datacenters
 ResolvedjcrespoT124795 codfw is in read only according to mediawiki
 ResolvedjcrespoT119642 Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases)

Event Timeline

Krinkle created this task.Feb 3 2016, 4:39 PM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added projects: codfw-rollout-Jan-Mar-2016, codfw-rollout.
Krinkle subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 3 2016, 4:39 PM
Krinkle added a subtask: T114398: Make and document DC switch-over scripts.Feb 3 2016, 4:40 PM
Krinkle added a subtask: T125510: Traffic Infrastructure support for Mar 2016 codfw rollout.
Krinkle added a subtask: T114271: Audit *all* datastores to see how DR failover is handled.
Krinkle added a subtask: T124670: Figure out and document the datacenter switchover process.Feb 3 2016, 4:42 PM
Krinkle triaged this task as High priority.Feb 3 2016, 5:00 PM
Krinkle set Security to None.
Krinkle added a subtask: T124795: codfw is in read only according to mediawiki.
JEumerus subscribed.Feb 3 2016, 5:11 PM
faidon subscribed.Feb 10 2016, 4:44 PM

Is there a point into tracking this with a task rather than codfw-rollout-Jan-Mar-2016? We've been collecting tasks in the latter, so far.

gerritbot subscribed.Mar 7 2016, 12:57 PM

Change 275443 had a related patch set uploaded (by Giuseppe Lavagetto):
realm: add $::master_dc hash

https://gerrit.wikimedia.org/r/275443

gerritbot added a project: Patch-For-Review.Mar 7 2016, 12:57 PM
gerritbot added a comment.Mar 7 2016, 4:56 PM

Change 275539 had a related patch set uploaded (by Giuseppe Lavagetto):
iegreview: use $parsoid_primary

https://gerrit.wikimedia.org/r/275539

jcrespo edited subtasks, added: T124699: Change configuration to make codfw db masters as the masters of all datacenters; removed: T124795: codfw is in read only according to mediawiki.Mar 8 2016, 2:24 PM
gerritbot added a comment.Mar 15 2016, 2:49 PM

Change 275443 merged by Giuseppe Lavagetto:
realm: add $::app_routes hash

https://gerrit.wikimedia.org/r/275443

gerritbot added a comment.Mar 16 2016, 8:18 AM

Change 275539 merged by Giuseppe Lavagetto:
iegreview: use $::parsoid_site

https://gerrit.wikimedia.org/r/275539

BBlack closed subtask T125510: Traffic Infrastructure support for Mar 2016 codfw rollout as Resolved.Mar 23 2016, 4:02 PM
Aklapper added a comment.Apr 9 2016, 8:05 AM

@Krinkle: Could you answer T125673#2015272 please? Thanks.

Volans closed subtask T124699: Change configuration to make codfw db masters as the masters of all datacenters as Resolved.Apr 20 2016, 1:21 PM
Krinkle closed this task as Invalid.Apr 21 2016, 3:52 AM
jcrespo closed subtask T124670: Figure out and document the datacenter switchover process as Resolved.Apr 28 2016, 9:53 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL