Change Details

It's a debugging feature that can go away after the whole goal is finished. In order to be able to debug and test the new infrastructure for jobs, support gradual job-by-job rollout and test the new infrastructure without disrupting old, proven to work, job queue implementation, we need to post events both to the old job queue and to the #eventbus . The easiest way to achieve that is to introduce a new hook `AfterJobSubmit`, listen to it in the #eventbus extension and try posting the `IJobSpecification` to #eventbus. Then we'd be able to experiment with new JobQueue without disrupting what's working now, and, once established the queue works for a particular job, switch the queues one-by-one. Since the hook will only be useful during the transition period only within WMF infrastructure, it should only be enabled in the particular queue we are using, `JobQueueRedis` and it would be clearly documented as `deprecated` from the very beginning.

In order to be able to develop, test, debug and stabilize the new kafka-based job pipeline without disrupting the currently working job processing, and also to be able to switch the jobs one-by-one to the new pipeline, starting from the least complex/important jobs, we need to be able to post jobs to the #eventbus simultaneously with posting them to the Redid-based JobQueue. It's a debugging feature that will be used during the transition period and it can go away when the transition is complete. There're several options that we can use: **Option 1: Create an `AfterJobPush` hook** This is the easiest way to achieve the result right now. The hook will be executed after the job has been posted to the `JobQueueRedis` receiving `IJobSpecification` as a parameter, #eventbus extension will subscribe to the hook and try posting the job to #eventbus, surrounding the serialization code heavily with try/catch and logging any serialization errors. Since the new hook will only be useful during the transition in the WMF production, it will be marked as `deprecated` right from the beginning, clearly documented as a temporary feature and removed later. Pros: - Less code right now, allows to start testing the serialization quicker - No hard dependency on `EventBus` extension being installed - The serialization/posting code can easily be transferred to the `JobQueueEventBus` implementation without changes - No need for `mediawiki-config` changes - Less likely to break existing pipeline Cons: - Doesn't test the `JobQueueEventBus` right away, only tests serialization and event posting - The hook is by default public, so someone might start using it for other purposes even though it will be documented as temporary feature **Option 2: Create a `JobQueue` delegate that posts to 2 separate JobQueues** Create a delegate that would construct two `JobQueue` implementation - the primary and the secondary one. All the method invocations on the delegate would be passed to the primary instance, why posting will be also delegated to the secondary queue. Since this feature is needed only during the transition period, it will be marked as `deprecated` right away and will be dominated as a temporary feature. Pros: - Makes us create and examine the `JobQueueEventBus` implementation earlier - Don't need changes to the existing `EventBus` extension that should be reverted afterwards - Less likely that someone would use the temporary feature Cons: - More non-trivial code is needed - More likely to disrupt the current job pipeline if some of the methods is not properly delegated - Requires changes to `mediawiki-config` - Creates a hard dependency on `EventBus` extension being installed.

In order to be able to develop, test, debug and stabilize the new kafka-based job pipeline without disrupting the currently working job processing, and also to be able to switch the jobs one-by-one to the new pipeline, starting from the least complex/important jobs, we need to be able to post jobs to the #eventbus simultaneously with posting them to the Redid-based JobQueue. It's a debugging feature that will be used during the transition period and it can go away afterwhen the whole goaltransition is finishedcomplete. In order to be able to debug and test the new infrastructure for jobs, support gradual job-by-job rollout and test the new infrastructure without disrupting old, proven to work, job queue implementation, we need to post events both to the old job queue and to the #eventbus .There're several options that we can use: The easiest way to achieve that is to introduce a new hook `AfterJobSubmit`, listen to it in the #eventbus extension and try posting the `IJobSpecification` to #eventbus. Then we'd be able to experiment with new JobQueue without disrupting what's working now, and, once established the queue works for a particular job, switch the queues one-by-one.**Option 1: Create an `AfterJobPush` hook** Since the hook will only be useful during the transition period only within WMF infrastructure,This is the easiest way to achieve the result right now. The hook will be executed after the job has been posted to the `JobQueueRedis` receiving `IJobSpecification` as a parameter, #eventbus extension will subscribe to the hook and try posting the job to #eventbus, surrounding the serialization code heavily with try/catch and logging any serialization errors. Since the new hook will only be useful during the transition in the WMF production, it will be marked as `deprecated` right from the beginning, clearly documented as a temporary feature and removed later. Pros: - Less code right now, allows to start testing the serialization quicker - No hard dependency on `EventBus` extension being installed - The serialization/posting code can easily be transferred to the `JobQueueEventBus` implementation without changes - No need for `mediawiki-config` changes - Less likely to break existing pipeline Cons: - Doesn't test the `JobQueueEventBus` right away, only tests serialization and event posting - The hook is by default public, so someone might start using it for other purposes even though it will be documented as temporary feature **Option 2: Create a `JobQueue` delegate that posts to 2 separate JobQueues** Create a delegate that would construct two `JobQueue` implementation - the primary and the secondary one. All the method invocations on the delegate would be passed to the primary instance, why posting will be also delegated to the secondary queue. it should only be enabled in the particular queue we are usingSince this feature is needed only during the transition period, `JobQueueRedis` and it would be clearly documented as `deprecated` from the very beginning.it will be marked as `deprecated` right away and will be dominated as a temporary feature. Pros: - Makes us create and examine the `JobQueueEventBus` implementation earlier - Don't need changes to the existing `EventBus` extension that should be reverted afterwards - Less likely that someone would use the temporary feature Cons: - More non-trivial code is needed - More likely to disrupt the current job pipeline if some of the methods is not properly delegated - Requires changes to `mediawiki-config` - Creates a hard dependency on `EventBus` extension being installed.