Page MenuHomePhabricator

Set mediawiki's read-only mode automatically when database masters are detected to have read_only=1
Closed, DuplicatePublic

Description

The current system to set mediawiki in read-only mode requires one deploy to set it and another to leave it. While the current system is ok (being manually possible to set it by a configuration change), operations is shifting to a more automatic system.

One option would be to orchestrate that configuration directly from confd. Deploying now such change takes around ~2 minutes per change, increasing the visible user impact (while actual failover time tends to tale only 1-2 seconds).

Another, more complex option to implement in a scalable way and reliable way, but that would reduce down time is to automatically set it if a write fails because it is in read only mode or the master fails to respond at all. I know mediawiki can detect such a state because when it happens it can log such an error (but continues writing). I believe also some error handling is already implemented. By setting read only mode on master database read only mode, we would increase the agility of a server or datacenter failover steps (orchestration would set the servers automatically, which would itself notify mediawiki- with no deployments needed).

Of course, this should be setup in a way that this is not checked on every query, but that on error, it is set for all connections. The exact process can be discussed, this is why this is an RFC.

Event Timeline

Only adding Performance-Team because it is (loosely) related to multi-datacenter work and they could give feedback about the best way to proceed with this (may need a memcache or APC implementation?).

Danny_B renamed this task from [RFC] Set mediawiki's read-only mode automatically when database masters are detected to have read_only=1 to Set mediawiki's read-only mode automatically when database masters are detected to have read_only=1.May 19 2016, 9:25 AM
Danny_B added a project: Proposal.
Joe triaged this task as Medium priority.EditedMay 19 2016, 2:31 PM

I think the problem is, more in general: how can we pass values from a configuration system like etcd to MediaWiki?

The "logical" candidate would be confd, which has a few pretty annoying issues though.

I would suggest we give it a try first; the easiest thing would be to have a simple template translate etcd values to a php file on disk.

I would suggest we give it a try first

This is the point of this task- have a smaller scope than a "full confd-controlled" database config. with very clear actionables/goals instead of trying to do everything at once.

The scope here would be: having a "service" for each shard and datacenter that controls the "read-only" variable. However, can confd do something else than create templates or do we need custom code? MySQL config (unlike mediawiki) cannot be reloaded in a hot way, so we would need a code hook to apply the changes directly.

Aside from that, which I do not disagree, there are some mediawiki reliability issues that I want to approach at the same time here. Would mediawiki do the right thing if a master went down or would we just get a large amount of errors? Could job write execution be controlled at the same time, rather than having to kill those manually?

I am open to handle these 2 aspects (ops-orchestration and mediawiki-HA) on different tickets if you consider it preferable.