Our current implementation of ChangeDispatchCoordinator relies on global MySQL locks, which hogs connections to the master DB, and may lose locks when these connections are reset by a watchdog.
@aaron suggested to use Redis based locks instead.
When implementing ChangeDispatchCoordinator based on RedisLockManager, please take care to closely replicate the semantics of the existing SqlChangeDispatchCoordinator.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T108944 [Epic] Improve change dispatching | |||
| Resolved | Ladsgroup | T151681 DispatchChanges: Avoid long-lasting connections to the master DB | |||
| Resolved | Ladsgroup | T151993 Implement ChangeDispatchCoordinator based on RedisLockManager | |||
| Resolved | Andrew | T155042 Increase quota for wikidata-dev project | |||
| Resolved | Ladsgroup | T155190 Build an environment to test change dispatching using Redis-based locking | |||
| Declined | None | T155196 Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant | |||
| Resolved | Ladsgroup | T157308 Create test script for ChangeDispatchCoordinator |
Change 325640 had a related patch set uploaded (by Ladsgroup):
[very WIP] [Draft] [DNM] [I can't emphasize enough] Add RedisLockSqlChangeDispatchCoordinator
@aaron: Hey, I'm working on this but it seems that RedisLockManager (and LockManagers in general) doesn't have lock lookup or I couldn't find it. Can you clarify if lock managers have something equivalent to Database::lockIsFree ?
There is no analogous method. Maybe a non-blocking engageClientLock() call can replace the isClientLockUsed() call? Seems like chd_lock is used to determine whether to do a non-blocking check first before a blocking acquisition (which could race anyway, which I guess just effects runtime?).
Without looking at the code, it seems we should be able to do without Database::lockIsFree. It's nice to do a check first to avoid a more expensive attempt to acquire a lock if we already know that that is likely to fail, but in the end, all we need is an atomic way to try to grab a lock, and then know if we got it.
One thing that is important in this context is how stale locks are handled. Does redis know if the owner of a lock has died? Or an orphaned lock stay around forever? Do locks time out?
If the owner fatals, the lock will have to expire (the TTL depends on the LockManager instance config and/or whether the context is CLI or web).
Change 325640 merged by jenkins-bot:
[mediawiki/extensions/Wikibase] Add LockManagerSqlChangeDispatchCoordinator
For implementation details. Let's keep talking in T159828: Use redis-based lock manager for dispatchChanges on test.wikidata.org and T159826: Use redis-based lock manager in dispatch changes in production
Change 344750 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Document dispatchingLockManager option in $wgWBRepoSettings
It just occurred to me an extra reason to avoid using a db master- master failover is a relative frequent operation: it will happen every time the master mysql is upgraded, or when there is a datacenter failover (2 of those will happen on April/May)- probably there wasn't any safe guard to avoid issues when that happened.
Closing this particular task (which only holds some of the things we need to do in order to actually improve the situation considerably).