It's a debugging feature that can go away after the whole goal is finished.In order to be able to develop, test, debug and stabilize the new kafka-based job pipeline without disrupting the currently working job processing, and also to be able to switch the jobs one-by-one to the new pipeline, starting from the least complex/important jobs, we need to be able to post jobs to the #eventbus simultaneously with posting them to the Redid-based JobQueue. It's a debugging feature that will be used during the transition period and it can go away when the transition is complete.
There're several options that we can use:
**Option 1: Create an `AfterJobPush` hook**
This is the easiest way to achieve the result right now. The hook will be executed after the job has been posted to the `JobQueueRedis` receiving `IJobSpecification` as a parameter, #eventbus extension will subscribe to the hook and try posting the job to #eventbus, surrounding the serialization code heavily with try/catch and logging any serialization errors. Since the new hook will only be useful during the transition in the WMF production, it will be marked as `deprecated` right from the beginning, clearly documented as a temporary feature and removed later.
- Less code right now, allows to start testing the serialization quicker
- No hard dependency on `EventBus` extension being installed
- The serialization/posting code can easily be transferred to the `JobQueueEventBus` implementation without changes
- No need for `mediawiki-config` changes
- Less likely to break existing pipeline
- Doesn't test the `JobQueueEventBus` right away, only tests serialization and event posting
- The hook is by default public, so someone might start using it for other purposes even though it will be documented as temporary feature
**Option 2: Create a `JobQueue` delegate that posts to 2 separate JobQueues**
In order to be able to debug and test the new infrastructure for jobs,Create a delegate that would construct two `JobQueue` implementation - the primary and the secondary one. support gradual job-by-job rollout and test the new infrastructure without disrupting oldAll the method invocations on the delegate would be passed to the primary instance, proven to work,why posting will be also delegated to the secondary queue. job queue implementaSince this feature is needed only during the transition period, we need to post events both to the old job queue and to the #eventbus it will be marked as `deprecated` right away and will be dominated as a temporary feature.
The easiest way to achieve that is to introduce a new hook `AfterJobSubmit`, listen to it in the #eventbus extension and try posting the `IJobSpecification` to #eventbus. Then we'd be able to experiment with new JobQueue without disrupting what's working now, and, once established the queue works for a particular job, switch the queues one-by-one.Pros:
- Makes us create and examine the `JobQueueEventBus` implementation earlier
- Don't need changes to the existing `EventBus` extension that should be reverted afterwards
- Less likely that someone would use the temporary feature
Since the hook will only be useful during the transition period only within WMF infrastructure, it should only be enabled in the particular queue we are using, `JobQueueRedis` and it would be clearly documented as `deprecated` from the very beginning.Cons:
- More non-trivial code is needed
- More likely to disrupt the current job pipeline if some of the methods is not properly delegated
- Requires changes to `mediawiki-config`
- Creates a hard dependency on `EventBus` extension being installed.