In order to be able to develop, test, debug and stabilize the new kafka-based job pipeline without disrupting the currently working job processing, and also to be able to switch the jobs one-by-one to the new pipeline, starting from the least complex/important jobs, we need to be able to post jobs to the Event-Platform simultaneously with posting them to the Redid-based JobQueue. It's a debugging feature that will be used during the transition period and it can go away when the transition is complete.
There're several options that we can use:
Option 1: Create an AfterJobPush hook
This is the easiest way to achieve the result right now. The hook will be executed after the job has been posted to the JobQueueRedis receiving IJobSpecification as a parameter, Event-Platform extension will subscribe to the hook and try posting the job to Event-Platform, surrounding the serialization code heavily with try/catch and logging any serialization errors. Since the new hook will only be useful during the transition in the WMF production, it will be marked as deprecated right from the beginning, clearly documented as a temporary feature and removed later.
Pros:
- Less code right now, allows to start testing the serialization quicker
- No hard dependency on EventBus extension being installed
- The serialization/posting code can easily be transferred to the JobQueueEventBus implementation without changes
- No need for mediawiki-config changes
- Less likely to break existing pipeline
Cons:
- Doesn't test the JobQueueEventBus right away, only tests serialization and event posting
- The hook is by default public, so someone might start using it for other purposes even though it will be documented as temporary feature
Option 2: Create a JobQueue delegate that posts to 2 separate JobQueues
Create a delegate that would construct two JobQueue implementation - the primary and the secondary one. All the method invocations on the delegate would be passed to the primary instance, why posting will be also delegated to the secondary queue. Since this feature is needed only during the transition period, it will be marked as deprecated right away and will be dominated as a temporary feature.
Pros:
- Makes us create and examine the JobQueueEventBus implementation earlier
- Don't need changes to the existing EventBus extension that should be reverted afterwards
- Less likely that someone would use the temporary feature
Cons:
- More non-trivial code is needed
- More likely to disrupt the current job pipeline if some of the methods is not properly delegated
- Requires changes to mediawiki-config
- Creates a hard dependency on EventBus extension being installed.