Hmm, looks like Dragonfly is able to outcompete Redis in terms of performance and resource usage. I'm interested in trying that. Not sure if that can be run on trove, but I can set it up on a VPS project myself!

Edit: on second thought, nevermind. We don't need a large scale thing like dragonfly. Redis provides good enough performance on a single threaded instance already.

Edit 2: also saw https://redis.com/blog/redis-architecture-13-years-later/, reaffirming that Redis is good enough.

The redis trove database can be named wikiauthbot2 since dashes are discouraged.

@0xDeadbeef unfortunately we don't currently support Redis as a database type in Trove. @Andrew do you think it's something we could enable?

The Redis for Toolforge service might work fine, I don't think we ever prune the data intentionally but I might be wrong, I will ask. What happens if the tool loses the data? If it simply causes the users to have to re-link their account, maybe it's not that bad?

Finally, we can also consider a small Cloud VPS instance where you could install and manage a Redis instance yourself.

Per https://github.com/wikimedia/operations-puppet/blob/ee02ed8031f246dd4f309bb7e83eeb9a8f285f51/modules/profile/manifests/toolforge/redis_sentinel.pp#L79-L80 and https://redis.io/docs/reference/eviction/#eviction-policies, it looks like the Redis for Toolforge instance will start deleting old keys once the maxmemory limit is reached. This is usually fine if applications don't want to use redis as a reliable persistent storage (its advertised as a useful cache service), but it would be a bad thing for this specific tool.

What happens if the tool loses the data? If it simply causes the users to have to re-link their account, maybe it's not that bad?

The users will have to re-link their account, yes, but it can be problematic because I'm not sure how much data are currently kept with that instance and unsure about the frequency the oldest keys get evicted. I would additionally have to write code that monitors when authentication data gets deleted and notify the users to authenticate again, since otherwise we'd just be silently losing track of them. The administration would also want to be able to retrieve information about who's behind a discord account or what discord accounts are linked to a specific wikimedia account, which would mean that making the data persistent would be necessary.

Happy to manage it myself, or use Trove, either would work.

In T358427#9591405, @fnegri wrote:

@0xDeadbeef unfortunately we don't currently support Redis as a database type in Trove. @Andrew do you think it's something we could enable?

I've tested redis with trove a bit and it's quite broken in the current code version. Best bet is probably for the project to run its own redis and make some kind of periodic backup dump to disk if it contains valuable/persistent data.

Best bet is probably for the project to run its own redis and make some kind of periodic backup dump

Agreed.

@0xDeadbeef I wonder if you prefer having a separate Cloud VPS project just for this, or if you'd like to bundle up several Discord-related bots you are managing (like T358337) in a single project?

Bundling would actually be nice. It would probably help with the bus factor since Legoktm (or any future collaborators) can access both if things happen. We can rename it to something like discordbots if possible, but I have not gotten into accessing the trove project yet so I'm unfamiliar with the technicalities behind it. Let me know if there's anything I can do! And thanks.

I think renaming a project is complicated, I will create a new project discordbots and delete the existing loggerdiscordbot project. Once the new project is created, you'll be able to select it from the project dropdown in Horizon, manage the Trove database under "Database->Instances", and the VM for Redis under "Compute->Instances".

Mentioned in SAL (#wikimedia-cloud) [2024-03-04T14:56:32Z] <dhinus> delete project "loggerdiscordbot" in favor of new project "discordbots" T358337,T358427

Thank you!