This doesn't sound good:
| August 21
|--
| {F9141847 height=150}
Current: <https://grafana.wikimedia.org/dashboard/db/job-queue-health?refresh=1m&orgId=1>
### Mitigation
* 6898d06f8f Disable rebound CDN purges for backlinks in HTMLCacheUpdateJob. @aaron, https://gerrit.wikimedia.org/r/373705
In MediaWiki 1.27 (2015) we introduced `$wgCdnReboundPurgeDelay` (more information at <https://www.mediawiki.org/wiki/Manual:$wgCdnReboundPurgeDelay>). This means that effectively every purge is actually two purges, spaced out by a few seconds, to account for race conditions between database replication and cache population. However, the common case of recursive updates (e.g. from templates) is already delayed through the JobQueue, so we can omit the rebound purge in that case.
**→** Fixing this improved job execution throughput, and decreased job queue growth.
* 70d1bc0091 Make workItemCount() smarter for htmlCacheUpdate/refreshLinks. @aaron, <https://gerrit.wikimedia.org/r/373325>
To avoid pressure on CDN and database, job execution can be throttled via <https://www.mediawiki.org/wiki/Manual:$wgJobBackoffThrottling>. The placeholder jobs for recursive updates (e.g. for templates) were being counted as 1, which needlessly subjected them to throttling.
**→** Fixing this improved job execution throughput.
* 56ed177f1 Reduce batch size in WikiPageUpdater. @Ladsgroup, <https://gerrit.wikimedia.org/r/#/c/373551/>
**→** Fixing this improved job execution throughput.
* cb7c910ba Fix old regression in HTMLCacheUpdate de-duplication. @aaron, <https://gerrit.wikimedia.org/r/373979>
Following a refactor in mid-2016, the duplication logic for recursive updates started to be miscalculated (based on current time, instead of time from the initial placeholder/root job).
**→** Fixing this significantly decreased job queue growth.
| Dashboard: Job Queue Helath
|--
| {F9232210 height=150} {F9232209 height=200}
| Job queue size down from 10M to a steady ~2M. Before the regression it was a steady between 100K -1M.
| [Dashboard: Varnish stats](https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?orgId=1&from=1503729100778&to=1504135040719)
|--
| {F9232784 height=250}
| Purge rate from production Varnish servers reduced by **2-3X**, from 75-100K/s to ~30K/s.
| [Dashboard: Job Queue Rate for htmlCacheUpdate](https://grafana.wikimedia.org/dashboard/db/job-queue-rate?panelId=2&fullscreen&orgId=1&var-Job=htmlCacheUpdate&from=now-90d&to=now)
|--
| {F9232855 height=200}
| Queue rate of `htmlCacheUpdate` back to normal. Deduplication/Superseding optimisation is now working. Execution speed has increased.
See also:
* {T124418}