Page MenuHomePhabricator

Wikibase: Increase batch size for HTMLCacheUpdateJobs triggered by repo changes.
Closed, DeclinedPublic


Currently, the number of pages purged by a singe HTMLCacheUpdateJob is limited to 20, by the wikiPageUpdaterDbBatchSize setting. The low number was set a few months ago because this setting used to also control the batch size for the expensive RefreshLinksJob, see rOMWCc29fae9270d1: Reduce wikiPageUpdaterDbBatchSize to 20.

However, as per rEWBA8c96687c2e46: Allow batch sizes for different jobs to be defined separately., we no longer apply batching for RefreshLinksJob, and wikiPageUpdaterDbBatchSize is deprecated in favor of purgeCacheBatchSize and recentChangesBatchSize.

Since we have problems with the a number of HTMLCacheUpdateJob being pushed to the job queue (see e.g. T173710#3699426), we should try to increase the batch size for this job, perhaps to 100 or 200.

The value for recentChangesBatchSize could perhaps also be increased, though in the light of rEWBAb00fac476e80: Quick fix for flooding of the RC table we should be careful here. A larger batch size may be help or hurt.

Side note: with T178806: Wikibase: Batch HTMLCacheUpdateJobs across changes implemented, it would become more common for jobs to actually hit the batch size. Right now, I believe many (most?) HTMLCacheUpdateJobs that wikibase generates do not hit that limit.

Event Timeline

daniel created this task.Oct 23 2017, 3:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 23 2017, 3:30 PM
daniel added a project: SRE.EditedOct 23 2017, 4:37 PM

@Joe To clarify, these parameters are for ops to tweak. I can guess at good values, but this is really your department, I think. From the Wikidata perspective, the batch size for purge jobs doesn't matter, we just want them to work. You can set it to whatever you think is sensible.

daniel updated the task description. (Show Details)Oct 23 2017, 4:38 PM
thiemowmde triaged this task as Low priority.Oct 27 2017, 12:35 PM
Krinkle closed this task as Declined.Jul 11 2018, 3:04 AM

Closing out as this seems specific to the old Redis-based JobQueue and JobRunner that are no longer as of last week. Please re-open if it still applies.