Page MenuHomePhabricator

Figure out a plan to move forward with regarding Redis License changes
Open, MediumPublic

Description

Problem

On March 20, 2024, Redis announced that, starting with Redis 7.4, they will no longer be releasing Redis under the 3clause BSD license but they would be rather be dual licensing under the Redis Source Available License and the Server Side Public License. The details are in https://redis.com/blog/redis-adopts-dual-source-available-licensing/

Neither of the 2 licenses above are Open Source licenses as per the OSI definition. In case it isn't obvious, in Redis they are acutely aware of that and even the naming of the licenses reflects that. The OSI has very clearly stated it for SSPL in https://opensource.org/blog/the-sspl-is-not-an-open-source-license and RSALv2 is in the same spirit anyway

Given the "Freedom and Open Source" Guiding Principle as stated in https://foundation.wikimedia.org/wiki/Resolution:Wikimedia_Foundation_Guiding_Principles

we need to figure out how to react/respond to this license change.

Timeline

We got time, even Debian sid has Redis 7.2 right now (https://packages.debian.org/sid/redis), which is released under the 3 clause BSD license. My expectation is that they will eventually drop Redis from the distribution though. A very rough estimation says anything between 2 and 4 years before we have to urgently deal with the problem.

Current usages

Production Realm

Currently there are apparently 3 main use cases of Redis in production

WMCS Realm

  • Toolforge offers a Redis service
  • Quarry
  • redisLockManager in Beta cluster

Possible directions forward

Disclaimer

This task has been filed on the day following the announcement. As such, there are no clear paths forward and we need first to do some investigation. That being said, it isn't that difficult to discern the rough shape of the possible directions. I am not currently advocating for any, this is just a list and not even exhaustive.

Fork of Redis

We either expect someone to fork redis or fork it ourselves.

In the former case, this would be just a repetition of what happened with Elasticsearch and Opensearch and a path we 've been down before.

The latter case is a lot more involved and would require substantial investment on our side. This will be exacerbated by the fact that we would be, de facto, assuming that role of for a lot of people on the globe.

Emerging forks

Stop using Redis altogether

In this scenario, we evaluate all usages of Redis one-by-one, figuring out replacement solutions and eventually stop using Redis before the need to use versions past (and including) 7.4. There's many details on this one as some of the tools and code paths using Redis aren't controlled by the Movement (see Gitlab), others are abandoned and others are properly maintained.

Continue using Redis, accepting it's not Open Source anymore

This is arguably possible under the Guiding Principle per

As an organization, we strive to use open source tools over proprietary ones, although we use proprietary or closed tools (such as software, operating systems, etc.) where there is currently no open-source tool that will effectively meet our needs.

However, this includes the burden of proving that no open source tool that effectively meets our needs exists. It also includes the burden of avoiding critical dependencies on proprietary code or services for maintaining a largely functionally equivalent fork. which is expressively stated a paragraph above the quoted one.

I should note that in the case that a fork of Redis happens after we decide to go down some path that follows under this direction, we are effectively forced to fallback to scenarios that fall under the "Fork of Redis" direction.

Investigation

Before any plans are made to follow any of the rough directions outlined above, we 'll need to perform an evaluation of our usages of Redis.

Event Timeline

https://github.com/Snapchat/KeyDB already existed as a fork. https://github.com/Snapchat/KeyDB/issues/798 was filed explicitly asking about the upstream licensing changes:

This project will remain with its original licensing.

KeyDB is currently based off Redis 6 primarily because of a lack of features that we needed from 7. However if we have enough time I do want to sync to the last true open source release of Redis.

brennen added subscribers: Jelto, brennen.

For GitLab: I think we currently run the bundled Redis in their Omnibus package. In that case, the easiest thing here will be if GitLab upstream replaces that on their own. I think it's also possible to configure it to use our own instance, if GitLab doesn't do anything about it and we have a viable fork available.

cc: @Jelto

In MediaWiki (as deployed at WMF), there exists 1 use of Redis, which is during file uploads via LockManager. There is nothing particularly special or unique Redis offers there, we just haven't bothered moving it to anything else since it works.

There's a proposal at T161749 for unifying the dc-local mutex lock use cases in MediaWIki behind a shared interface, and configuring prod to consistently use PoolCounter for it (with BagOStuff as default, in practice that woudl be SQL, Memcached or Redis for third-parties).

In MediaWiki (as deployed at WMF), there exists 1 use of Redis, which is during file uploads via LockManager. There is nothing particularly special or unique Redis offers there, we just haven't bothered moving it to anything else since it works.

There's a proposal at T161749 for unifying the dc-local mutex lock use cases in MediaWIki behind a shared interface, and configuring prod to consistently use PoolCounter for it (with BagOStuff as default, in practice that woudl be SQL, Memcached or Redis for third-parties).

Worth considering using mainstash (x2) instead of redis, filename lock is not that important if you ask me. The move to commons extension already does its locking via mainstash.

If we decide to move away from x2/mariadb, it'd be transparent to mw.

In MediaWiki (as deployed at WMF), there exists 1 use of Redis, which is during file uploads via LockManager.

Same deal for the beta cluster (at least, LockManager / T340908 seemed to be the only breakage when Redis was removed by accident).

Also used in Vagrant for a bunch of things (job queue, maybe sessions).

PhpRedis is getting behind KeyDB with #2466 and I encouraged them along that path with a small PR of my own. I think all we need to do for now in MediaWiki is update our documentation to say that KeyDB is supported.

As API Gateway is nowadays owned by serviceops, adding the serviceops project tag to open API Gateway tasks tagged with the deprecated/archived "Platform Team Initiatives (API Gateway)" tag at https://phabricator.wikimedia.org/project/profile/4321/, as part of Phabricator Housekeeping.

LWN has an article titled "The race to replace Redis". I am not going to link directly as it is LWN subscriber only content but I can summarize (note I am pasting links in their entirety on purpose) the "Forks and alternatives" section:

As far as distros go (just adding these for completeness, given we have taken a very long time ago the decision to use Debian)

My 2, operationally minded, cents says to wait for the dust to settle a little bit before moving forward with any plan. We got time and I have my doubts regarding all 4 different forks surviving.

My 2, operationally minded, cents says to wait for the dust to settle a little bit before moving forward with any plan. We got time and I have my doubts regarding all 4 different forks surviving.

+1 It is too soon for us to catch any of those waves.

Additionally, my 2 cents here say that, any future solution, should have the prospect of providing High Availability (or something as close to it as possible). This, of course, might include a combination of components (see mcrouter+memcached), which is OK.

Forgive the drive-by comment, but at the 6-month anniversary of this ticket, it might be worth checking how our upstream production applications (such as gitlab, netbox etc) are handling this change, if it all. For example, I noticed that netbox-docker is now using valkey .