Status quo

The status quo is that we tend to acquire locks in whatever we have handy in the same component. This has some short-term benefits in terms of learning curve. For example, if a component X is primary reading-writing things from a database with Rdbms, then it can use that same database to perform any locks it might need to coordinate work between requests. Likewise, if a component Y is primarily computing data and caching it in memcached with BagOStuff, it might as well use memcached locks to coordinate writes between requests when needed.

There are however limitations to this, which leads to cases where we currently reach for other solutions. Let's start with a list of components we have at the moment inside MediaWiki that are capable of providing locks:

1. Rdbms (MySQL, Postgres, Sqlite): Our database interface is primarly used for reading and writing rows in our various database tables. Our supported database types all support named locks, and so IDatabase::lock in Rdbms provide access to this capability in a way that naturally translates to this native database capability.
   • Failsafe: Deny all locks, throws exception. This is appropriate as "low tier" solution, with relatively low demand/traffic, where you can afford to enforce uniqueness by lock as a hard requirement (i.e. hundreds per minute, not thousands per second).
   • Capacity: Low.
   • Availability: High.
2. BagOStuff (Null, Memcached, Redis, APCu): Our cache interface for storing ephemeral key-value data. These typically don't have an explicit "lock" feature, but, it is easy to emulate this by storing a temporary key like foobar-my-lock with a value of 1 and checking whether that exists. And so BagOStuff::lock provides this behind an easy to use interface.
   • Failsafe: Grant all locks. This is a "high tier" solution (can handle large loads, with best-effort guruantees). The failure mode, including the default when no cache backend is provided, is to grant all locks (i.e. no keys appear to exist).
   • Capacity: High.
   • Availability: Medium.
3. PoolCounter (Null, Redis, poolcounterd): This is a dedicated MediaWiki component specifically for locks and locks only. It was developed in 2009 by @tstarling following the Michael Jackson effect. The locks are kept in memory in one of several independent central poolcounterd processes. It is designed for the highest uptime and load capacity. Locks are hashed/mapped to 1 of 3 hosts, with fallback to the other two in order in case of downtime. It protects the servers against an overload from demands for specific unavailable content, by ensuring server capacity is never allocated all to the same task, thus making sure capacity remains available for other page views. Despite being in the critical path for page views, we have to trust it enough that we can failsafe to rendering an error (or stale content) instead of best-effort rendering something anyway. This is made possible through the service being implemented in a few hundred lines of stable C code, with no other responsibilities. The caveat here is htat PoolCounter is an optional dependency and so by default we use a Null placeholder where by all locks are granted. This makes sense for the purpose of PoolCounter, since it is something you won't need until you're pretty large in scale (e.g. Wikipedia). However, lack of a default means we shy away from this as our go-to solution. We only use as extra layer for things important to Wikipedia.
   • Failsafe: Deny. Render stale content or display an error.
   • Capacity: Highest.
   • Availability: Highest.
4. LockManager (Files, Redis, Memcached): This is a dedicated MediaWiki component specifically for locks and locks only. It was developed specifically for MediaWiki FileBackend and is used as part of the multimedia upload process (e.g. when uploading photos to Wikimedia Commons). It was created in 2011 by @aaron when MediaWiki evolved to store files beyond a local or NFS-mounted hard drive for photos in favour of a dedicated Swift cluster (wikitech: Media storage § History). Before that point, FileBackend would lock file paths using file operations directly on the local/NFS disk. This is coded in a way that achieved both high availability and high guarantees. That it has a sensible default (based on local files in the hard drive), but has an interchangable Redis implementation as well. The Redis implementation uses quorums so that it isn't merely best-effort based on the responsiveness and uptime of a single Redis server. Instead of sharding keys across the cluster (like you would for caching), it uses quorums to increase uptime and offer higher guruantees. The another notable aspect of LockManager is that it allows acquiring multiple locks at once in a way that reliably grants you all or nothing.
   • Failsafe: Deny all locks, if a quorum can't be reached (e.g. less than 2/3 redis-lock hosts are up and responding in time)
   • Capacity: Medium.
   • Availability: Medium.

Examples when we can't use "whatever we have at hand":

• Reducing load on a service by using locks in the same service, is ineffective. For example, you can't reduce connections to a primary database host through a lock on that same database (whereby only the first request to get the lock would connect and do the work). If you use that same primary database to store the locks, then you still end up establishing too many primary DB connections. In this situation we tend to grab something else semi-random. For example, you might reach for memcached/BagOStuff to create a best-effort lock instead.
  • Example: ResourceLoader/DependencyStore. Before ResourceLoader writes data, it uses the "main cache" (usually memcached via BagOStuff) to acquire a lock and thus ignore and debounce any redundant attempts from other requests around the same time.
  • Downside: The "main cache" is optional in MediaWiki (default: CACHE_NONE) and so by default this wil not actually de-duplicate the work for third party setups. In practice there is generally always a reasonable place to put locks, even in a plain install. But knowing which one to pick at runtime is non-trivial.

Problem statement

Currently, if high-performance blocking locks are desired for tasks (such as cache regeneration de-duplication), the only option is PoolCounter.

In the sub-usecase of mutexes (vs semaphores), a broader set of backends could be used than those that could implement the *full* set of PoolCounter features.

For example, MySQL named locks could handle the exclusive regeneration (PC's acquireForMe case) case of PoolCounter, but not the case of multiple threads blocking on the same key (e.g. X threads can be "doing Y" at the same time).

The case of de-duplicating parsing work on edit fits into the former case (e.g. waiting on any prior edit stash request to finish for that edit) but not the later.

Proposed solution

For simplicity and flexibility, it would be nice if a new interface could be introduced for interacting with these locks. This would allow for the application to select any appropriate backend.

This would let the codebase fall back to things likes mysql named locks (when possible) but use faster, multi-DC appropriate things PoolCounter, if installed/available.

Currently, such logic would require a config variable and some if/else logic for the named lock vs PoolCounter case. A common interface for all native blocking locks would simplify this kind of scenario. Demanding PoolCounter is unnecessarily constraining and falling back to polling loops is suboptimal.

• https://gerrit.wikimedia.org/r/332950 ("lockmanager: Add InterruptMutexManager interface")
• https://gerrit.wikimedia.org/r/332951 ("Implement InterruptMutexManager methods in PoolCounter_Client")