Switchover of the application servers to codfw
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Joe
	Jan 25 2016, 4:52 PM

Description

For switching over the appservers, we need to do the following:

Make it easy to switch the configuration of mediawiki to have a different primary datacenter (see T114273)
Correct all the things that are eqiad-only in mediawiki-config (there are many, introduced after we got it on par...)
Perform a quick sanity check of performance of a sample of pages in the two DCs
Check the amount of resources available in the various clusters - codfw was sized as a slightly reduced eqiad at the time, but we did change some ratios in eqiad in the meantime.

See also: Why they're doing this https://meta.wikimedia.org/wiki/Tech/Server_switch_2016

Details

	Subject	Repo	Branch	Lines +/-
	Use the logical redis definition for GettingStarted.	operations/mediawiki-config	master	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Invalid	None	T125673 Switch over from Eqiad to Codfw as primary datacentre
Resolved	jcrespo	T124670 Figure out and document the datacenter switchover process
Resolved	Krinkle	T124671 Switchover of the application servers to codfw
Resolved	None	T124879 Check the memcached and redis (sessions) configuration and functionality in codfw
Resolved	jcrespo	T124697 Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming
Resolved	jcrespo	T125386 Clarify how mysql dumps will be architectured during codfw failover
Resolved	jcrespo	T124795 codfw is in read only according to mediawiki
Resolved	Joe	T126242 Reduce the number of appservers we're using in eqiad
Resolved	• Cmjohnson	T129060 mw1026-69 are shut down and should be physically decommissioned
Resolved	• Cmjohnson	T133770 mw1070-89 and mw1121-30 are shut down and should be physically decommissioned
Resolved	Joe	T129062 Reorganize mw2* servers across clusters

Event Timeline

Joe created this task.Jan 25 2016, 4:52 PM

Joe raised the priority of this task from to Medium.

Joe updated the task description. (Show Details)

Joe added projects: codfw-rollout-Jan-Mar-2016, codfw-rollout, SRE.

Joe added subscribers: faidon, Aklapper, Joe.

Joe updated the task description. (Show Details)Jan 25 2016, 5:01 PM

Joe set Security to None.

Joe updated the task description. (Show Details)Jan 26 2016, 10:32 AM

Change 266481 had a related patch set uploaded (by Giuseppe Lavagetto):
Use the logical redis definition for GettingStarted.

https://gerrit.wikimedia.org/r/266481

gerritbot added a project: Patch-For-Review.Jan 26 2016, 10:37 AM

Change 266481 merged by jenkins-bot:
Use the logical redis definition for GettingStarted.

https://gerrit.wikimedia.org/r/266481

jcrespo added a subtask: T124697: Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming.Feb 1 2016, 12:40 PM

matmarex reopened subtask T126242: Reduce the number of appservers we're using in eqiad as Open.Feb 8 2016, 6:32 PM

JJMC89 subscribed.Mar 1 2016, 7:28 PM

Xaosflux subscribed.Mar 1 2016, 10:49 PM

(Possibly silly question alert: in case things went wrong, would this be reflected on http://status.wikimedia.org/ , or would that site be down as well? )

zhuyifei1999 subscribed.Mar 2 2016, 4:33 PM

Glaisher subscribed.Mar 2 2016, 4:59 PM

In T124671#2080234, @Elitre wrote:

(Possibly silly question alert: in case things went wrong, would this be reflected on http://status.wikimedia.org/ , or would that site be down as well? )

status.wikimedia.org is hosted outside of our servers and as such will not go down while this takes place.

In T124671#2080234, @Elitre wrote:

(Possibly silly question alert: in case things went wrong, would this be reflected on http://status.wikimedia.org/ , or would that site be down as well? )

Neither, really. That page will stay up but as a monitor it's quite naive and won't catch a lot of the things that may not go wrong with this switchover. So, for instance, if we stay read-only for a prolonged period of time it would be a failure on our part but would not be reflected on that status page.

Liuxinyu970226 subscribed.Mar 4 2016, 9:41 AM

Joe closed subtask T124879: Check the memcached and redis (sessions) configuration and functionality in codfw as Resolved.Mar 7 2016, 8:17 AM

Joe changed the status of subtask T126242: Reduce the number of appservers we're using in eqiad from Open to Stalled.Mar 7 2016, 10:38 AM

Joe created subtask T129062: Reorganize mw2* servers across clusters.Mar 7 2016, 10:56 AM

Joe closed subtask T129062: Reorganize mw2* servers across clusters as Resolved.Mar 7 2016, 12:53 PM

Ricordisamoa subscribed.Mar 7 2016, 8:53 PM

IKhitron subscribed.Mar 7 2016, 9:45 PM

RandomDSdevel awarded a token.Mar 7 2016, 11:02 PM

Trizek subscribed.Mar 8 2016, 3:31 PM

Trizek unsubscribed.

Trizek-WMF subscribed.Mar 8 2016, 3:31 PM

Matthewrbowker subscribed.Mar 10 2016, 4:54 PM

Akeron subscribed.Mar 10 2016, 6:34 PM

Nemo_bis added a project: WMF-General-or-Unknown.Mar 10 2016, 8:50 PM

Nemo_bis subscribed.

IKhitron unsubscribed.Mar 10 2016, 8:58 PM

#3 (and #4, partially -for mediawiki "simple" user HTTP requests and dbs) is being done at "careful speed", but more extensively at T124697. That ticket does not include testing things like the jobqueue, specialpages update, search, parsoid, etc.

Raymond subscribed.Mar 11 2016, 1:57 PM

Luke081515 subscribed.Mar 11 2016, 4:09 PM

Mbch331 subscribed.Mar 11 2016, 7:58 PM

• Whatamidoing-WMF updated the task description. (Show Details)Mar 14 2016, 4:49 PM

tomasz unsubscribed.Mar 18 2016, 5:27 PM

jcrespo closed subtask T124697: Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming as Resolved.Mar 31 2016, 4:10 PM

Joe changed the status of subtask T126242: Reduce the number of appservers we're using in eqiad from Stalled to Open.Apr 1 2016, 6:41 AM

Krinkle moved this task from Backlog to In Progress on the codfw-rollout-Jan-Mar-2016 board.Apr 21 2016, 3:02 PM

Krinkle closed this task as Resolved.Apr 21 2016, 3:07 PM

Krinkle claimed this task.

Krinkle moved this task from In Progress to Done on the codfw-rollout-Jan-Mar-2016 board.

• Elitre awarded a token.Apr 21 2016, 3:40 PM