| jijiki | |
| Nov 22 2019, 6:03 PM |
| F31133354: response.png | |
| Nov 22 2019, 6:03 PM |
| F31133353: varnish.png | |
| Nov 22 2019, 6:03 PM |
Today since around 13:14 UTC, we have observed an increased latency in our avg and 95th percentile
https://grafana.wikimedia.org/d/5E7tdiGWz/xxxx-effie?orgId=1&refresh=30s&panelId=13&fullscreen
At ~18:36 there was another spike in long-tail latency, but then, latency seemed to return to 'normal':
https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1&from=1574443410296&to=1574450056679
Found this task only now, but see also T238973: Appservers rising GET latency might have triggered LVS pages
Probably, given I'm investigating on Saturday. But I think there is hope this can be mitigated. I'll keep the task posted.
Just to clarify - the situation got worrisome only this morning, when latencies skyrocketed and the issue became user-visible. I'm not sure the two issue are the same, but for convenience I'm going to keep using this task.
Restarting php-fpm on the affected servers did solve the issue. I decided against doing deeper debugging before restarting the fleet because of the urgency of the fix (the problem became user-visible).