**Currently thet state: **The "dispatch" process for wikidata / wikibase is kicked off from a cron job and maintenance script.
**Problem**
To simplify things we want to avoid the requirement to run this additional cron script, instead containing the whole process within jobWikibase users (including WIkibase developers) do not want to have to run extra maintenance scripts.
(dis[patching is part of what is covered in this docs) https://doc.wikimedia.org/Wikibase/master/php/md_docs_topics_change-propagation.htmlWMF SREs do not want to manage extra cron jobs (they cause complexity during in cross data centre work)
This is interesting for 3rd parties, as they also do not want to have to run a separate cron script**Docs**
This is also interesting for WMF SREs as they do not want to have to manage extra cron jobs for maintenance scripts (it makes data center switchovers harder[The Repository dispatching (script, db tables, jobs)](https://doc.wikimedia.org/Wikibase/master/php/md_docs_topics_change-propagation.html)
In production this cron jobs can be seen at https://github.com/wikimedia/puppet/blob/e1e13a59de3021afaa43c31745abbe348a93017d/modules/profile/manifests/mediawiki/maintenance/wikidata.pp[The Client receiving its notification event / job](https://wmde.github.io/wikidata-wikibase-architecture/systems/WikibaseClient/06-Runtime_View.html#entity-change-notifications)
Rough Idea:
* Every edit schedules a delayed DispatchTriggerJob. That job is completely generic and holds no info at all, so DispatchTriggerJob of this kind are the same. This means that new jobs get ignored if there is already an older job waiting for execution.
* (option a) DispatchTriggerJob would poll the changes table, as we do now, and dispatch any pending changes to the most lagged wiki(s). This means that passes for long tail wikis will often end up doing nothing. If "doing nothing" is quick enough, we could simply go and look at the next wiki, until some minimum number of changes has been processed, or some maximum time has been exceeded.
* (option b) DispatchTriggerJob would take the next batch of changes, and send notifications for all of them to the interested wikis. That means that each pass has to (potentially) push to all wikis, which may take quite long.
* If there are still pending jobs or wikis to service, DispatchTriggerJob schedules another (delayed?) DispatchTriggerJob before it exists. How many new triggeres should be scheduled? We need to avoid starvation, but also prevent explosive growth of the number of trigger jobs.
**Acceptance criteria🏕️🌟:**
[] No maintenance script / cron job needs to be run for the dispatching process to work
[] De-duplication should be used to avoid creating too many unnecessary jobs
[] Documentation should be updated
**Notes:**
When this task is tackled it should be taken in mind that some refactoring will likely make sense, such as {T256208} (but this is also tracked and prioritized speseparately.
This should be gradually deployed, and this could possibly be done in a couple of different ways:
- Per environment: beta, test, production
- Per client wiki (or group of wikis) within each environment: group1, group2, (everything except enwiki), enwiki, commonswiki
Overall performance of these jobs will be dictated by the job queue processing, which is controlled by WMF SREs and service ops?
We have a general performance requirement of "The dispatching process for Wikidata should not be slower than it currently is"
The code to be deployed from this ticket likleylikely won't have a big impact in performance, though the configuration of the processing of jobs may, and this would need to be figured out with serviceops.
In Wikidata production this cron jobs can be seen at https://github.com/wikimedia/puppet/blob/e1e13a59de3021afaa43c31745abbe348a93017d/modules/profile/manifests/mediawiki/maintenance/wikidata.pp