Page MenuHomePhabricator

Design and implement async LDAP operations
Closed, ResolvedPublic

Description

Various parts of IDM will initiate LDAP changes (since multiple people can be logged into the IDM in multiple sessions) and those requests will need to be handed off to a task/job queue for processing. Some requests will also be dependent on each other (e.g. creating a group and then adding a given user to it) and there needs to be some workflow for handling errors as well. So if e.g. a user makes an edit to their preferences, then the IDM would validate the input and launch an LDAP job to effect the change and poll (up to a given timeout) for the successful or failed completion of the job.

We should evaluate existing schedulers (such as Celery or RQ) for their fitness to the task or otherwise write something custom.

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
OpenSLyngshede-WMF
ResolvedNone
OpenNone
Opentaavi
OpenNone
ResolvedMarostegui
ResolvedAndrew
ResolvedMarostegui
ResolvedAndrew
DeclinedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedLadsgroup
DuplicateNone
Resolved Bstorm
DeclinedNone
Resolvedtaavi
ResolvedJdforrester-WMF
DeclinedNone
Openjijiki
OpenNone
OpenFeatureNone
StalledFeatureNone
OpenFeatureSLyngshede-WMF
OpenNone
OpenAndrew
OpenSLyngshede-WMF
ResolvedABran-WMF
Resolvedtaavi
OpenNone
OpenSLyngshede-WMF
ResolvedPRODUCTION ERRORTgr
OpenNone
Resolvedbd808
Resolvedyuvipanda
Resolvedbd808
Resolvedbd808
Resolvedbd808
Opentaavi
Resolvedtaavi
DeclinedNone
OpenNone
ResolvedSLyngshede-WMF
ResolvedSLyngshede-WMF

Event Timeline

SLyngshede-WMF changed the task status from Open to In Progress.Nov 2 2022, 10:30 AM
SLyngshede-WMF claimed this task.

We'll attempt to build using RQ and the Django RQ module. RQ supports basic job queuing, as well as job dependencies and the ability to get job status.
Other than supporting the expected use-cases RQ have the advantage of utilizing Redis, rather than RabbitMQ. While more powerful and feature complete for queuing, protocol support topology, RabbitMQ have the disadvantage of being more difficult to manage and debug in failure scenarios. Redis is in contrast easier to manage and inspect in case of issues.

Basic proof-of-concept for queuing have been done of simple queues.
Remaining is the job dependency and job status. These are supported by RQ directly, but it's unclear if Django RQ supports these out of the box. If not we can simply fallback to standard RQ functionality.

We're going with Django-RQ as it's simpler and does not require Celery.