Page MenuHomePhabricator

Ensure replica DB in labs is read-only
Closed, DuplicatePublic

Description

wikiadmin@deployment-db03[deploymentwiki]> SELECT @@global.read_only;
+--------------------+
| @@global.read_only |
+--------------------+
|                  0 |
+--------------------+
1 row in set (0.00 sec)

wikiadmin@deployment-db04[deploymentwiki]> SELECT @@global.read_only;
+--------------------+
| @@global.read_only |
+--------------------+
|                  0 |
+--------------------+
1 row in set (0.00 sec)

deployment-db04 is the replica, with deployment-db03 as its master, but it's not read-only. Because of a bug in MediaWiki (see T183242: DB handles obtained with DB_REPLICA should not allow writes), writes were being done directly to the replica, which caused inconsistencies and broke replication.

It should not have been possible for this replica server to be writable. Out of paranoia, I checked in production, and none of the replica DBs in production are writable.

Details

Event Timeline

Change 399188 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/core@master] [MCR] Fix SqlBlobStore using DB_REPLICA for writes

https://gerrit.wikimedia.org/r/399188

Change 399188 merged by jenkins-bot:
[mediawiki/core@master] [MCR] Fix SqlBlobStore using DB_REPLICA for writes

https://gerrit.wikimedia.org/r/399188

I've manually set deployment-db04 to read-only, but there should be a structural solution in place to prevent this from happening again. We somehow do manage to do this in production: every single production replica server is read-only, and I'm pretty sure there are guarantees of some kind for that.