Page MenuHomePhabricator

Allow changeDispatcher to run with different configuration for different groups of wikis
Closed, InvalidPublic


ChangeDispatcher has various options governing batch size, dispatch interval, lock retention, etc. These settings are designed to allow a tradeoff between optimization through batching and acceptably low delays until changes are processed on the client wiki. This tradeoff however strongly depends on the size of the client wiki (or rather, the number of entities used on the client wiki, and the number of edits to those entities on the repo).

However, with the current setup, we have to find a compromise between good settings for large wikis and good settings for small wikis, leading to situations like T171263: Wikidata Dispatcher and Job Queue is overflowed.

To allow us to optimize for both large and small client wikis, we should be able to run dispatchChanges cron jobs with different settings for different groups of wikis. To achieve this, we could add a chd_group column to wb_change_dispatch, and add an option --group to dispatchChanges, which filters by the value in that new DB field.

Event Timeline

So, how would you optimize for the large wikis specifically which would not work (well) for small wikis?

Lydia_Pintscher subscribed.

Closing this to remove tasks from the board that havn't gotten any attention in a long time.