codfw is in read only according to mediawiki
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	Jan 26 2016, 5:51 PM

Description

Should it be in read only? Or only the servers should be? Right now, their masters point to eqiad masters, but that is a security concern and a performance issue (allowing cross-datacenter queries)
If it should, we can set the masters as local masters.
Alternatively, we can remove the mediawiki read-only option and set only the databases as read-only
Problem with read only is that maintenance scripts have to be run from the local datacenter

Details

	Subject	Repo	Branch	Lines +/-
	Prepare db-codfw.php for a live deployment	operations/mediawiki-config	master	+258 -61

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Invalid	None	T125673 Switch over from Eqiad to Codfw as primary datacentre
Resolved	jcrespo	T124670 Figure out and document the datacenter switchover process
Resolved	Krinkle	T124671 Switchover of the application servers to codfw
Resolved	jcrespo	T124697 Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming
Resolved	Joe	T124672 Check the redis (jobqueue) configuration in codfw
Resolved	RobH	T126453 codfw: (2) servers for redis jobrunners
Resolved	elukey	T129178 rack/setup/deploy rdb200[5-6]
		Unknown Object (Task)
		Unknown Object (Task)
		Unknown Object (Task)
Resolved	Joe	T126470 Enable Redis cross-dc replication
Resolved	Krinkle	T124673 Figure out how to migrate the jobqueues
Resolved	aaron	T129250 Ensure post-send handlers check and respect read-only-mode
Resolved	Krinkle	T129258 Ensure maintainers of long-running scripts on terbium expect downtime for switchover
Resolved	elukey	T123675 Reinstall redis servers (Job queues) with Jessie (NOTE: rdb1002 is special and is excluded!)
Resolved	elukey	T128730 Sudden increase in NOTICE events from hhvm while trying to de-pool rdb1003 for maintenance
Resolved	Joe	T129317 Dedicate 1/2 codfw jobrunners to gwtoolset jobs
Resolved	Joe	T126235 Have a strategy to switch restbase to use services in the appropriate datacenter
Resolved	jcrespo	T129236 Improve documentation about database switchover
Resolved	Volans	T124699 Change configuration to make codfw db masters as the masters of all datacenters
Resolved	aaron	T88445 MediaWiki active/active datacenter investigation and work (tracking)
Resolved	Joe	T114273 Cleanup active-DC based MW config code and make it more robust and easy to change
Resolved	jcrespo	T124795 codfw is in read only according to mediawiki

Event Timeline

jcrespo created this task.Jan 26 2016, 5:51 PM

jcrespo raised the priority of this task from to Needs Triage.

jcrespo updated the task description. (Show Details)

jcrespo added projects: MediaWiki-Configuration, DBA, SRE, codfw-rollout-Jan-Mar-2016.

jcrespo added subscribers: jcrespo, Krenair, Joe.

Restricted Application added a project: codfw-rollout. · View Herald TranscriptJan 26 2016, 5:51 PM

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

Dereckson subscribed.Jan 26 2016, 5:51 PM

My personal recommendation is to make it 100% read only, point to local masters, and force maintenance from the active datacenter always. cc Release-Engineering-Team

phab doesn't auto-add projects like that anymore

jcrespo: how do we make it 100% read-only? is there an easy way to do that?

I agree that we should stop using cross-dc masters in that config, it was just a naive first approximation of what we wanted to do, I guess.

@Joe, it is a configuration change. Should be "trivial". I opened this ticket because it has more repercussions than that- I think there is a legitimate need for being in read only mode (for example, there is an outage that is creating general lag, or there is maintenance), but we want to allow writes to the database using mediawiki mechanisms (to fix that issue). I would like feedback from deployers/tin/mira/terbium users, and a more general solution may require mediawiki changes (?). That way we do not realize too late this is needed.

Change 267659 had a related patch set uploaded (by Jcrespo):
Delete eqiad masters from codfw configuration and add db weights

https://gerrit.wikimedia.org/r/267659

gerritbot added a project: Patch-For-Review.Feb 1 2016, 11:38 AM

Krinkle added a parent task: T88445: MediaWiki active/active datacenter investigation and work (tracking).Feb 3 2016, 5:01 PM

Krinkle added a parent task: T125673: Switch over from Eqiad to Codfw as primary datacentre.

jcrespo added a parent task: T114273: Cleanup active-DC based MW config code and make it more robust and easy to change.Feb 24 2016, 5:54 PM

I'm ok with leaving the master databases pointing to the original ones but that is 1) more reasons to create a split brain (editing one file but not the other), 2) how to avoid accidental cross-datacenter writes.

Having one file, or only one file for masters may be the way to go.

jcrespo claimed this task.Feb 25 2016, 7:46 PM

jcrespo triaged this task as High priority.

jcrespo moved this task from Triage to In progress on the DBA board.

jcrespo added a parent task: T124699: Change configuration to make codfw db masters as the masters of all datacenters.Mar 2 2016, 4:54 PM

jcrespo moved this task from Backlog to In Progress on the codfw-rollout-Jan-Mar-2016 board.Mar 8 2016, 2:04 PM

jcrespo removed a parent task: T125673: Switch over from Eqiad to Codfw as primary datacentre.Mar 8 2016, 2:24 PM

jcrespo added a parent task: T124697: Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming.Mar 9 2016, 7:05 PM

Change 267659 merged by jenkins-bot:
Prepare db-codfw.php for a live deployment

https://gerrit.wikimedia.org/r/267659