Page MenuHomePhabricator

codfw is in read only according to mediawiki
Closed, ResolvedPublic

Description

  • Should it be in read only? Or only the servers should be? Right now, their masters point to eqiad masters, but that is a security concern and a performance issue (allowing cross-datacenter queries)
  • If it should, we can set the masters as local masters.
  • Alternatively, we can remove the mediawiki read-only option and set only the databases as read-only
  • Problem with read only is that maintenance scripts have to be run from the local datacenter

Details

Related Gerrit Patches:
operations/mediawiki-config : masterPrepare db-codfw.php for a live deployment

Related Objects

StatusAssignedTask
InvalidNone
Resolvedjcrespo
ResolvedKrinkle
Resolvedjcrespo
ResolvedJoe
ResolvedRobH
Resolvedelukey
ResolvedJoe
ResolvedKrinkle
Resolvedaaron
ResolvedKrinkle
Resolvedelukey
Resolvedelukey
ResolvedJoe
ResolvedJoe
Resolvedjcrespo
ResolvedVolans
Openaaron
ResolvedJoe
Resolvedjcrespo

Event Timeline

jcrespo created this task.Jan 26 2016, 5:51 PM
jcrespo raised the priority of this task from to Needs Triage.
jcrespo updated the task description. (Show Details)
jcrespo added subscribers: jcrespo, Krenair, Joe.
Restricted Application added a project: codfw-rollout. · View Herald TranscriptJan 26 2016, 5:51 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

My personal recommendation is to make it 100% read only, point to local masters, and force maintenance from the active datacenter always. cc Release-Engineering-Team

phab doesn't auto-add projects like that anymore

Joe added a comment.Jan 27 2016, 11:03 AM

jcrespo: how do we make it 100% read-only? is there an easy way to do that?

I agree that we should stop using cross-dc masters in that config, it was just a naive first approximation of what we wanted to do, I guess.

@Joe, it is a configuration change. Should be "trivial". I opened this ticket because it has more repercussions than that- I think there is a legitimate need for being in read only mode (for example, there is an outage that is creating general lag, or there is maintenance), but we want to allow writes to the database using mediawiki mechanisms (to fix that issue). I would like feedback from deployers/tin/mira/terbium users, and a more general solution may require mediawiki changes (?). That way we do not realize too late this is needed.

Change 267659 had a related patch set uploaded (by Jcrespo):
Delete eqiad masters from codfw configuration and add db weights

https://gerrit.wikimedia.org/r/267659

I'm ok with leaving the master databases pointing to the original ones but that is 1) more reasons to create a split brain (editing one file but not the other), 2) how to avoid accidental cross-datacenter writes.

Having one file, or only one file for masters may be the way to go.

jcrespo claimed this task.Feb 25 2016, 7:46 PM
jcrespo triaged this task as High priority.
jcrespo moved this task from Triage to In progress on the DBA board.

Change 267659 merged by jenkins-bot:
Prepare db-codfw.php for a live deployment

https://gerrit.wikimedia.org/r/267659

jcrespo closed this task as Resolved.Mar 10 2016, 7:09 PM
jcrespo added a subscriber: Krinkle.

Fixed with @Krinkle's merge.