Page MenuHomePhabricator

Allow changeDispatcher to run with different configuration for different groups of wikis
Closed, InvalidPublic

Description

ChangeDispatcher has various options governing batch size, dispatch interval, lock retention, etc. These settings are designed to allow a tradeoff between optimization through batching and acceptably low delays until changes are processed on the client wiki. This tradeoff however strongly depends on the size of the client wiki (or rather, the number of entities used on the client wiki, and the number of edits to those entities on the repo).

However, with the current setup, we have to find a compromise between good settings for large wikis and good settings for small wikis, leading to situations like T171263: Wikidata Dispatcher and Job Queue is overflowed.

To allow us to optimize for both large and small client wikis, we should be able to run dispatchChanges cron jobs with different settings for different groups of wikis. To achieve this, we could add a chd_group column to wb_change_dispatch, and add an option --group to dispatchChanges, which filters by the value in that new DB field.

Event Timeline

daniel created this task.Aug 12 2017, 8:11 PM
Restricted Application added subscribers: PokestarFan, Aklapper. · View Herald TranscriptAug 12 2017, 8:11 PM
hoo updated the task description. (Show Details)Aug 13 2017, 3:26 AM

So, how would you optimize for the large wikis specifically which would not work (well) for small wikis?

Lydia_Pintscher closed this task as Invalid.Jan 24 2019, 2:43 PM
Lydia_Pintscher added a subscriber: Lydia_Pintscher.

Closing this to remove tasks from the board that havn't gotten any attention in a long time.