Change Details

So, the backend purging queues in esams are way behind. On the one node I'm staring at the most, there are currently about 87 million backlogged purge requests, which is probably somewhere in the ballpark of 10 hours of lag time. The backlog is in the local daemon on the esams hosts themselves (so this isn't a network issue with delivering the purges to the hosts over the WAN), so the culprit is likely the ATS daemon consuming them slowly. This is text@eqiad GETs vs PURGEs the past week. You can see GETs have the usual organic pattern, and PURGEs are fairly spiky as we normally see. {F31726687} Whereas with text@esams, we see a curious pattern to the PURGE traffic, seems like it's being load-limited and somewhat-recovering when organic traffic is low overnight: {F31726689} <edited here>: Those purge rates are from the frontend varnish, but the frontend varnish PURGE queue doesn't receive entries until they've traversed the backend one, which is backlogged, so that's why we still see the effect there.

Purging queues are often backlogged in all datacenters, and in esams they are persistently backlogged. (Other locations tend to recover on their own, but esams is so loaded this doesn't happen there.) {F31756624} https://grafana.wikimedia.org/d/wBCQKHjWz/vhtcpd?orgId=1&from=now-7d&to=now&var-datasource=esams%20prometheus%2Fops&var-cluster=All&var-worker=Purger0&fullscreen&panelId=4 Work is ongoing in {T249583} to build and ship a newer, more scalable `purged` daemon to issue purges in parallel instead of serially. There's also {T250205} to lower the number of them, because as-is it's quite excessive.

SoPurging queues are often backlogged in all datacenters, the backend purging queues in esams are way behindand in esams they are persistently backlogged. On the one node I'm staring at the most, there are currently about 87 million backlogged purge requests(Other locations tend to recover on their own, which is probably somewhere in the ballpark of 10 hours of lag time. The backlog is in the local daemon on the esams hosts themselves (so this isn't a network issue with delivering the purges to the hosts over the WAN), so the culprit is likely the ATS daemon consuming them slowly.but esams is so loaded this doesn't happen there.) This is text@eqiad GETs vs PURGEs the past week. You can see GETs have the usual organic pattern, and PURGEs are fairly spiky as we normally see.{F31756624} https://grafana.wikimedia.org/d/wBCQKHjWz/vhtcpd?orgId=1&from=now-7d&to=now&var-datasource=esams%20prometheus%2Fops&var-cluster=All&var-worker=Purger0&fullscreen&panelId=4 {F31726687} Whereas with text@esams, we see a curious pattern to the PURGE traffic, seems like it's being load-limited and somewhat-recovering when organic traffic is low overnight: {F31726689} <edited here>: Those purge rates are from the frontend varnishWork is ongoing in {T249583} to build and ship a newer, but the frontend varnish PURGE queue doesn't receive entries until they've traversed the backend one, which is backloggedmore scalable `purged` daemon to issue purges in parallel instead of serially. There's also {T250205} to lower the number of them, so that's why we still see the effect therebecause as-is it's quite excessive.