Page MenuHomePhabricator

Support for master/slave FileBackend distinction in FileBackendMultiwrite
Closed, ResolvedPublic

Description

Typically, modules that use FileBackend will want to read from the closest cluster and write to the master cluster (or all, depending on implementation). The 'latest' flag (which already is in use) could force reads to use the master (or cross-DC quorum read ect).

One way to do this is by making MultiWriteFileBackend use the 'latest' flag in the same way as BagOStuff::READ_LATEST for ReplicatedBagOStuff. This works if the master cluster is in the one DC, which the other cluster(s) replicated from it. On the other hand, a Swift setup using a global cluster spread over the DCs (https://swiftstack.com/blog/2013/07/02/swift-1-9-0-release/) would not use this setup. Instead, callers could just use SwiftFileBackend directly, which proper read/write affinity settings set in the swift proxy server config. I assume we will at least start off with master/slave though before trying the global stuff.

Additionally, deferring writes to the non-master backends should be a feature. This would handle updates for the 99% case, and a tool like swift-repl can fix any missed updates to assure consistency within hours.

Event Timeline

aaron created this task.Sep 15 2015, 10:24 PM
aaron claimed this task.
aaron raised the priority of this task from to Normal.
aaron updated the task description. (Show Details)
aaron added projects: Availability, Epic.
aaron added subscribers: Krinkle, jcrespo, Glaisher and 12 others.

Change 239641 had a related patch set uploaded (by Aaron Schulz):
Added readAffinity flag to FileBackendMultiWrite

https://gerrit.wikimedia.org/r/239641

aaron updated the task description. (Show Details)Sep 19 2015, 11:11 PM
aaron set Security to None.

Change 239646 had a related patch set uploaded (by Aaron Schulz):
Added replication=async option to FileBackendMultiWrite

https://gerrit.wikimedia.org/r/239646

Change 239641 merged by jenkins-bot:
Added readAffinity flag to FileBackendMultiWrite

https://gerrit.wikimedia.org/r/239641

Joe added a comment.Sep 21 2015, 2:16 PM

@aaron once this is out we can reconfigure our mediawiki setup to use this, or would we need to modify the code further?

aaron added a comment.Sep 21 2015, 7:04 PM

The two patches above are basically it, code-wise. The rest is just some wmf-config/ changes getting swiftrepl running to cleanup inconsistencies in the background.

aaron updated the task description. (Show Details)Sep 21 2015, 7:07 PM
aaron updated the task description. (Show Details)Sep 21 2015, 7:10 PM
aaron removed a project: Epic.
aaron moved this task from Backlog to Doing on the Availability board.Sep 24 2015, 6:01 AM
aaron moved this task from Doing to Blocked on the Availability board.

Change 239646 merged by jenkins-bot:
Added replication=async option to FileBackendMultiWrite

https://gerrit.wikimedia.org/r/239646

aaron renamed this task from Use proxy server or class for master/slave FileBackend distinction to Support for master/slave FileBackend distinction in FileBackendMultiwrite.Oct 2 2015, 6:24 AM
aaron closed this task as Resolved.

Leaving T91869 for the config/wmf aspects.