High insertion rate of ParsoidCachePrewarmJob causes substantial backlog, causing prewarm jobs to be delayed by up to two hours and more.
It seems likely that the high intertion rates are caused by template edits that invalidate the cached rendering of many highly frequented pages. However, we were not able to identify any specific template edits that would have caused the spikes.
We suspect that the issue is amplified by a stampede effect on pages that are highly frequented but slow to parse. Stampede protection and deduplication may help.
NOTE: ParsoidCachePrewarmJob get scheduled on page view whenever a page is viwed and the cached output of the old parser is detected to be stale. The old parser uses PoolCounter for stampede protection - if we fail to get a lock, we'll serve stale output, and procede to schedule a ParsoidCachePrewarmJob. This may happen numberous times before the old parser has finiashed rendering.
NOTE: ParsoidOutputAccess does not use PoolCounter, so several ParsoidCachePrewarmJobs for the same page may be executing concurrently, re-parsing the same page.