With the ever growing number of cookbooks and cookbook runs, we're now at a point in which we start to need to be able to limit the amount of specific cookbooks that can be run in parallel.
In some cookbooks some steps require that other cookbooks run are not performing the same step at the same time. In other cases we might want to limit the number of parallel runs.
Those locks should naturally be distributed and have a TTL with a cleanup of stale locks in place.
Requirements
- As a cookbook developer I want to be able to say:
- No more than 3 parallel run of this cookbook can be running at any given time
- because the usual time this cookbooks take to run is N minutes, I want to set the TTL for the lock at say 2xN minutes
- This specific line of code (or method) can be run only once at any given time, no parallelism
- As a cookbook user:
- I want to be able to bypass the locking mechanism and run the cookbook in "unsafe" mode from the locks point of view for emergency reasons or because the locking backend has an outage
I want to know which cookbooks can lock one another.[out of scope]- If I encounter a lock, I want to know the user who holds the lock, the cumin host, how much time the lock has left,
and whether the cookbook is actively running or waiting at a user prompt[out of scope, has nothing to do with the locking feature]. - If my cookbook is holding the lock, I want to know the above information as well.
If my cookbook is waiting at a prompt, I want to be paged so I can complete the prompt.[out of scope, has nothing to do with the locking feature]I need this information to be available throughout my cookbook run, via CLI or dashboard.[it's available in the console and the logs, and there is no need to know about it, it's just a contention prevention mechanism, and depending on the settings multiple concurrent runs will be available for specific locks.]
- As a Spicerack developer:
- Each cookbook should set a lock for itself with a default reasonable concurrency limit and TTL, easily customizable for each cookbook.
- I want to check if existing libraries exists that do what's needed and implement it only if necessary
- All the libraries must be in Debian
- The backend where the locks are stored is not important as long as it's reliable and distributed
Existing libraries
After some research for Python libraries that implement a distributed locking mechanism with configurable concurrency/parallelism and a TTL, I didn't find any that were even close to the set of features needed here.
So if we need to implement one we should first decide which backend we want to use for it.
Locks backend
Given the current infrastructure at the WMF it seems natural to pick one of the existing backends that are easily usable for locking:
- Poolcounter
- PRO: in house, supports queue size (concurrency) for the locks, has a Python library (actually more than one)
- CON: requires a TCP connection to be kept open for each lock for the whole duration of the lock and some cookbooks last a week or more. It's also a cross-DC connection in some cases, to ensure the distributed locking.
- etcd
- PRO: already used in Spicerack via conftool, has native support for basic locks (even better with APIv3)
- CON: the APIv3 library for Python has not been released in 2 years and has 61 commits in master not officially released, some of them needed to fully support our cluster (DNS SRV record support has 2 PRs, none merged and multi-endpoint support is merged in master but not released). We could though use the APIv2 library (preferred) or the HTTP GRPC gateway one (less preferred).
- Redis
- CON: the Redlock Python libraries seems to be mostly unmaintained
- CON: its usage at the WMF has declined in recent times, might not be the smartest move to use it for this.
Given the above it would seem natural to me to pick etcd as backend, to be decided with which library/API but most likely the APIv2 python library for now, to be replaced with the APIv3 one if/once available.
Implementation proposal
- Use etcd as backend
- Pick a namespace for the keys and setup the ACLs for it (proposed name: /spicerack_locks
- Use the API2 for now
- Use the very simple native support for exclusive locks in etcd APIv2 to acquire the lock to write a more complex object that has the additional metadata needed to implement the feature. Decide how many native locks to use:
- Just one with a hardcoded name to be used for all write actions performed by Spicerack that manage locks
- Three different ones, one for each different key prefix (the automatic lock creation at every cookbook run (basically between START and END), spicerack modules custom locks, cookbooks custom locks)
- Many, one for each different lock to create
- To acquire a spicerack lock first the native simple lock will be acquired, then a JSON object will be read/written from/to etcd to set the lock. If the lock cannot be created because it has reached its size limit it will retry for few times with some sleeps in between.
- An example object written to etcd will be of the form (the comments are added here to better explain the fields):
key: '/spicerack_locks/cookbooks/sre.hosts.decommission' value: { '4e2677c7-541a-4f9e-afbb-cfdb69c440a1': { # UUID of the single lock acquired, auto-generated 'concurrency': 5, # How many parallel runs are allowed 'ttl': 120, # After how many seconds the lock should be considered stale and garbage collected 'created': '2023-07-17 10:50:35.974419', # The UTC datetime of when the lock was acquired 'owner': 'user@host', # The username and hostname where the cookbook that took the lock is running } }
- Alternatively it's possible to use the etcd's native TTL support making each UUID based lock a key in etcd with TTL inside the lock name's directory and each time traverse the directory to get all the existing locks. This should be done while holding a write lock to avoid race conditions at least until we can use the API v3 and use transactions.
- From the API point of view there will be a context manager method acquired(key: str, *, concurrency: int, ttl: int) -> None to be used with the with ... statement plus an acquire(key: str, *, concurrency: int, ttl: int) -> str and release(lock_id: str) -> None methods for standalone use
- At each cookbook run Spicerack will automatically acquire the lock for the key /spicerack_locks/cookbooks/$COOKBOOK_NAME
- Each cookbook can also create additional locks for specific steps getting a lock object via the usual accessor mechanism: spicerack.lock(). The key for the locks created this way will have a prefix /spicerack_locks/custom/$LOCK_NAME.
- Spicerack modules that will implement locking for some of their steps will use locks with a prefix /spicerack_locks/$MODULE_NAME/ (e.g. /spicerack_locks/spicerack.dhcp/).
I'll send shortly some CR to gerrit with a draft of the above proposal.