Change Details

After a new revision is saved, secondary data, like link tables, is updated asynchronously via `DeferredUpdates`. Deferred updates are executed out-of-band, after the transaction that updated the primary data (revision meta-data, etc) is complete. Depending on setup and situation, such updates may even be pushed to the job queue, and may not be processed until several minutes later. External tools that keep track of edits on the wiki, by polling RecentChanges or using some kind of life feed, can get stale data because of this. E.g. a tool that wants to replicate the category graph would query the `categorylinks` table (either directly or via the API) whenever a category page was edited. But the `categorylinks` table may not have been updated yet; the external graph cannot be kept up to date. The Wikidata Query Service is affected by this, regarding the `page_links` table, see {T145712}. There should be a mechanism for external tools to be notified when an edit has been fully processed. ----- Possible solution: for each edit, store the number of pending update jobs in a new field of the recentchanges table. When each update job completes, it decrements that counter. Entries in the recentchanegs table that have a non-zero count of pending updates can then be ignored when desired. Alternatively, defer the entry to recentchanges until all other updates have completed. This would require a guaranteed order of execution for jobs, though.

### Problem When a revision is created, there are secondary data updates we perform (such as link tables, categories etc.). These updates can be deferred via DeferredUpdates or JobQueue. The event for such revision is published to external consumers (e.g. via RCFeed, EventStreams, API:RecentChanges polling etc.) before all secondary data updates are applied. The problem is that this creates problems when consumers need to react to an event. As they have no means to know when to start reacting to it and and when the related changes to the database are complete from the API perspective. ### Original description After a new revision is saved, secondary data, like link tables, is updated asynchronously via `DeferredUpdates`. Deferred updates are executed out-of-band, after the transaction that updated the primary data (revision meta-data, etc) is complete. Depending on setup and situation, such updates may even be pushed to the job queue, and may not be processed until several minutes later. External tools that keep track of edits on the wiki, by polling RecentChanges or using some kind of life feed, can get stale data because of this. E.g. a tool that wants to replicate the category graph would query the `categorylinks` table (either directly or via the API) whenever a category page was edited. But the `categorylinks` table may not have been updated yet; the external graph cannot be kept up to date. The Wikidata Query Service is affected by this, regarding the `page_links` table, see {T145712}. There should be a mechanism for external tools to be notified when an edit has been fully processed. ----- Possible solution: for each edit, store the number of pending update jobs in a new field of the recentchanges table. When each update job completes, it decrements that counter. Entries in the recentchanegs table that have a non-zero count of pending updates can then be ignored when desired. Alternatively, defer the entry to recentchanges until all other updates have completed. This would require a guaranteed order of execution for jobs, though.

### Problem When a revision is created, there are secondary data updates we perform (such as link tables, categories etc.). These updates can be deferred via DeferredUpdates or JobQueue. The event for such revision is published to external consumers (e.g. via RCFeed, EventStreams, API:RecentChanges polling etc.) before all secondary data updates are applied. The problem is that this creates problems when consumers need to react to an event. As they have no means to know when to start reacting to it and and when the related changes to the database are complete from the API perspective. ### Original description After a new revision is saved, secondary data, like link tables, is updated asynchronously via `DeferredUpdates`. Deferred updates are executed out-of-band, after the transaction that updated the primary data (revision meta-data, etc) is complete. Depending on setup and situation, such updates may even be pushed to the job queue, and may not be processed until several minutes later. External tools that keep track of edits on the wiki, by polling RecentChanges or using some kind of life feed, can get stale data because of this. E.g. a tool that wants to replicate the category graph would query the `categorylinks` table (either directly or via the API) whenever a category page was edited. But the `categorylinks` table may not have been updated yet; the external graph cannot be kept up to date. The Wikidata Query Service is affected by this, regarding the `page_links` table, see {T145712}. There should be a mechanism for external tools to be notified when an edit has been fully processed. ----- Possible solution: for each edit, store the number of pending update jobs in a new field of the recentchanges table. When each update job completes, it decrements that counter. Entries in the recentchanegs table that have a non-zero count of pending updates can then be ignored when desired. Alternatively, defer the entry to recentchanges until all other updates have completed. This would require a guaranteed order of execution for jobs, though.