Page MenuHomePhabricator

codfw is in read only according to mediawiki
Closed, ResolvedPublic

Description

  • Should it be in read only? Or only the servers should be? Right now, their masters point to eqiad masters, but that is a security concern and a performance issue (allowing cross-datacenter queries)
  • If it should, we can set the masters as local masters.
  • Alternatively, we can remove the mediawiki read-only option and set only the databases as read-only
  • Problem with read only is that maintenance scripts have to be run from the local datacenter

Related Objects

StatusSubtypeAssignedTask
InvalidNone
Resolvedjcrespo
ResolvedKrinkle
Resolvedjcrespo
ResolvedJoe
ResolvedRobH
Resolvedelukey
ResolvedJoe
ResolvedKrinkle
Resolvedaaron
ResolvedKrinkle
Resolvedelukey
Resolvedelukey
ResolvedJoe
ResolvedJoe
Resolvedjcrespo
ResolvedVolans
OpenNone
ResolvedJoe
Resolvedjcrespo

Event Timeline

jcrespo raised the priority of this task from to Needs Triage.
jcrespo updated the task description. (Show Details)
jcrespo added subscribers: jcrespo, Krenair, Joe.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

My personal recommendation is to make it 100% read only, point to local masters, and force maintenance from the active datacenter always. cc Release-Engineering-Team

phab doesn't auto-add projects like that anymore

jcrespo: how do we make it 100% read-only? is there an easy way to do that?

I agree that we should stop using cross-dc masters in that config, it was just a naive first approximation of what we wanted to do, I guess.

@Joe, it is a configuration change. Should be "trivial". I opened this ticket because it has more repercussions than that- I think there is a legitimate need for being in read only mode (for example, there is an outage that is creating general lag, or there is maintenance), but we want to allow writes to the database using mediawiki mechanisms (to fix that issue). I would like feedback from deployers/tin/mira/terbium users, and a more general solution may require mediawiki changes (?). That way we do not realize too late this is needed.

Change 267659 had a related patch set uploaded (by Jcrespo):
Delete eqiad masters from codfw configuration and add db weights

https://gerrit.wikimedia.org/r/267659

I'm ok with leaving the master databases pointing to the original ones but that is 1) more reasons to create a split brain (editing one file but not the other), 2) how to avoid accidental cross-datacenter writes.

Having one file, or only one file for masters may be the way to go.

jcrespo triaged this task as High priority.
jcrespo moved this task from Triage to In progress on the DBA board.

Change 267659 merged by jenkins-bot:
Prepare db-codfw.php for a live deployment

https://gerrit.wikimedia.org/r/267659

jcrespo added a subscriber: Krinkle.

Fixed with @Krinkle's merge.