Page MenuHomePhabricator

Design and implement async LDAP operations
Closed, ResolvedPublic

Description

Various parts of IDM will initiate LDAP changes (since multiple people can be logged into the IDM in multiple sessions) and those requests will need to be handed off to a task/job queue for processing. Some requests will also be dependent on each other (e.g. creating a group and then adding a given user to it) and there needs to be some workflow for handling errors as well. So if e.g. a user makes an edit to their preferences, then the IDM would validate the input and launch an LDAP job to effect the change and poll (up to a given timeout) for the successful or failed completion of the job.

We should evaluate existing schedulers (such as Celery or RQ) for their fitness to the task or otherwise write something custom.

Event Timeline

SLyngshede-WMF changed the task status from Open to In Progress.Nov 2 2022, 10:30 AM
SLyngshede-WMF claimed this task.

We'll attempt to build using RQ and the Django RQ module. RQ supports basic job queuing, as well as job dependencies and the ability to get job status.
Other than supporting the expected use-cases RQ have the advantage of utilizing Redis, rather than RabbitMQ. While more powerful and feature complete for queuing, protocol support topology, RabbitMQ have the disadvantage of being more difficult to manage and debug in failure scenarios. Redis is in contrast easier to manage and inspect in case of issues.

Basic proof-of-concept for queuing have been done of simple queues.
Remaining is the job dependency and job status. These are supported by RQ directly, but it's unclear if Django RQ supports these out of the box. If not we can simply fallback to standard RQ functionality.

We're going with Django-RQ as it's simpler and does not require Celery.