Page MenuHomePhabricator
Create Task
Maniphest T199073

Perform a datacenter switchover (2018-19 Q1)
Closed, ResolvedPublic
Actions

Description

This is the tracking task for one of the Wikimedia Technology/Goals/2018-19 Q1.

Perform a datacenter switchover:

  • Successfully switch backend traffic (MediaWiki, Swift, RESTBase, and Parsoid) to be served from codfw with no downtime and reduced read-only time.
  • Serve the site from codfw for at least 3 weeks. Tentative dates are to switch over to codfw in the second half of September and switch back to eqiad after 3~4 weeks.
  • Refactor the switchdc script into a more re-usable automation library and update it to the newer switchover requirements.

Switchover

Services: Tuesday, September 11th 2018 14:30 UTC
Media storage/Swift: Tuesday, September 11th 2018 15:00 UTC
Traffic: Tuesday, September 11th 2018 19:00 UTC
Mediawiki: Wednesday, September 12th 2018: 14:00 UTC

Switchback:

Traffic: Wednesday, October 10th 2018 09:00 UTC (and maybe some prep work on Monday)
Mediawiki: Wednesday, October 10th 2018: 14:00 UTC
Services: Thursday, October 11th 2018 14:30 UTC
Media storage/Swift: Thursday, October 11th 2018 15:00 UTC

Details

SubjectRepoBranchLines +/-
mediawiki::maintenance: use mw_primary to enable/disable cronsoperations/puppetproduction+7 -6
graphite: use keepLastValue for thumbs drift alertoperations/puppetproduction+2 -2
graphite: alert when eqiad and codfw drift in number of thumbnailsoperations/puppetproduction+22 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedakosiarisT199073 Perform a datacenter switchover (2018-19 Q1)
 ResolvedMarosteguiT189107 DB meta task for next DC failover issues
 DeclinedMarosteguiT186503 Rebuild user_newtalk on db1052
 ResolvedSep 9 2018MarosteguiT189101 Deploy schema change for adding numeric primary key to wbqc_constraints table
 ResolvedMarosteguiT187089 Fix WMF schemas to not break when comment store goes WRITE_NEW
Restricted Task
Restricted Task
Restricted Task
Restricted Task
 ResolvedjcrespoT183249 Reimage and upgrade to stretch all dbproxies
 ResolvedMarosteguiT89737 Make several mediawiki table fields unsigned ints on wmf databases
 ResolvedMarosteguiT144010 Drop eu_touched in production
 ResolvedMarosteguiT201133 db1069 (x1 master) memory errors
 ResolvedMarosteguiT203548 Remove partitions from s7 masters (db1062 and db2040) for metawiki.pagelinks
 ResolvedakosiarisT203776 Successfully switch backend traffic (MediaWiki, Swift, RESTBase, Parsoid and services) to be served from codfw
 ResolvedMarosteguiT204127 Reclone db2054 and db2068
 ResolvedjcrespoT204311 Upgrade all core (mediawiki) database servers to mariadb 10.1
 Resolved• CmjohnsonT204302 db1062 management interface busy (no sessions allowed)
 Resolved• CmjohnsonT204493 db1061 management interface busy (no sessions allowed)
 ResolvedjcrespoT184805 Move some wikis to s5
 ResolvedVolansT199079 Refactor the switchdc script
 ResolvedJoeT199124 Remove all usages of $::mw_primary on puppet
 ResolvedMarosteguiT200509 Make sure multi-instance slaves page
 ResolvedNoneT163495 Mediawiki revision-related queries are failing with high rate for enwiki on codfw
 ResolvedMarosteguiT163351 codfw API slaves overloaded during the 2017-04-19 codfw switch
 ResolvedNoneT163413 Pool new server db2071
 ResolvedKrinkleT116557 AFComputedVariable::compute query timeouts
 ResolvedjcrespoT160984 Reduce max execution time of interactive queries or a better detection and killing of bad query patterns
 ResolvedhooT163544 Wikibase\TermSqlIndex::getMatchingTerms , Wikibase\TermSqlIndex::fetchTerms have bad performance after codfw failover
 ResolvedakosiarisT203777 Successfully switch backend traffic (MediaWiki, Swift, RESTBase, Parsoid and services) to be served from eqiad

Event Timeline

Volans created this task.Jul 9 2018, 8:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 9 2018, 8:33 AM
Marostegui added a subtask: T189107: DB meta task for next DC failover issues.Jul 9 2018, 8:33 AM
Marostegui subscribed.
jcrespo added a subtask: T163495: Mediawiki revision-related queries are failing with high rate for enwiki on codfw.Jul 11 2018, 7:59 AM
Krinkle renamed this task from Perform a datacenter switchover to Perform a datacenter switchover (2018-19 Q1).Jul 12 2018, 5:14 AM
MoritzMuehlenhoff triaged this task as High priority.Jul 13 2018, 2:58 PM
gerritbot subscribed.Aug 27 2018, 1:09 PM

Change 455553 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: alert when eqiad and codfw drift in number of thumbnails

https://gerrit.wikimedia.org/r/455553

gerritbot added a project: Patch-For-Review.Aug 27 2018, 1:09 PM
gerritbot added a comment.Aug 28 2018, 7:48 AM

Change 455553 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: alert when eqiad and codfw drift in number of thumbnails

https://gerrit.wikimedia.org/r/455553

• mobrovac mentioned this in T202982: Requests to MW 404 when on HTTPS.Aug 28 2018, 10:52 AM
gerritbot added a comment.Aug 28 2018, 10:54 AM

Change 455811 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: use keepLastValue for thumbs drift alert

https://gerrit.wikimedia.org/r/455811

gerritbot added a comment.Aug 28 2018, 10:57 AM

Change 455811 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: use keepLastValue for thumbs drift alert

https://gerrit.wikimedia.org/r/455811

gerritbot added a comment.Aug 29 2018, 4:31 PM

Change 456175 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki::maintenance: use mw_primary to enable/disable crons

https://gerrit.wikimedia.org/r/456175

gerritbot added a comment.Aug 30 2018, 12:13 AM

Change 456175 abandoned by Dzahn:
mediawiki::maintenance: use mw_primary to enable/disable crons

https://gerrit.wikimedia.org/r/456175

• MZMcBride subscribed.Aug 30 2018, 1:03 AM
tom29739 subscribed.Aug 30 2018, 1:51 AM
Rxy subscribed.Aug 31 2018, 4:28 PM
Stashbot mentioned this in T201858: Push thumbnails to both data centers.Sep 3 2018, 2:23 PM
Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2018-09-03T14:23:25Z] <godog> switch swift thumbnails active/active - T201858 T199073

akosiaris updated the task description. (Show Details)Sep 4 2018, 9:29 AM
Paladox subscribed.Sep 4 2018, 10:49 AM
Stashbot added a comment.Sep 10 2018, 9:30 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-10T09:30:32Z] <volans> starting execution of "cookbook sre.switchdc.mediawiki --live-test codfw eqiad" - T199073

akosiaris closed subtask T199124: Remove all usages of $::mw_primary on puppet as Resolved.Sep 10 2018, 9:48 AM
Stashbot added a comment.Sep 10 2018, 11:44 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-10T11:44:09Z] <volans> completed execution of "cookbook sre.switchdc.mediawiki --live-test codfw eqiad" - T199073

Jdforrester-WMF subscribed.Sep 11 2018, 6:32 PM
akosiaris closed subtask T203776: Successfully switch backend traffic (MediaWiki, Swift, RESTBase, Parsoid and services) to be served from codfw as Resolved.Sep 12 2018, 4:49 PM
• Fjalapeno subscribed.Sep 18 2018, 4:48 PM
Volans updated the task description. (Show Details)Sep 26 2018, 10:48 AM
Volans closed subtask T199079: Refactor the switchdc script as Resolved.Oct 5 2018, 1:03 PM
Marostegui closed subtask T189107: DB meta task for next DC failover issues as Resolved.Oct 11 2018, 5:22 AM
akosiaris closed subtask T203777: Successfully switch backend traffic (MediaWiki, Swift, RESTBase, Parsoid and services) to be served from eqiad as Resolved.Oct 11 2018, 3:38 PM
akosiaris closed this task as Resolved.Oct 11 2018, 4:10 PM
akosiaris claimed this task.
akosiaris updated the task description. (Show Details)

Successfully switched (with some aftermath and actionables but successfully nevertheless) to codfw and back per the subtasks, I am resolving this.

AndyRussG mentioned this in T206562: Delay in French mobile banners showing up in Banner Allocation.Oct 15 2018, 1:52 AM
greg closed subtask T163495: Mediawiki revision-related queries are failing with high rate for enwiki on codfw as Resolved.Nov 27 2018, 7:25 PM
Marostegui mentioned this in T200022: 2018 data center switchover: Move all the things over to codfw.Oct 19 2020, 10:58 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 19 2020, 11:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL