Change Details

We're seeing [[ https://grafana.wikimedia.org/d/000000143/navigation-timing?orgId=1&var-source=navtiming2&var-metric=responseStart&var-percentile=All&from=now-30d&to=now | a spike of latency across the board ]] from real users' Navigation Timing responseStart metric that seems to start around 2020-09-28: {F32370851} Is there ongoing maintenance, outage or work from #traffic that could explain this?

After seeing strong time-correlation between the response start regression and the rollout of Varnish 6 per DC for Eqsin on 2020-09-25: {F32377027} And for Esams on 2020-09-29: {F32377025} Varnish was rolled back to its previous version on 2020-10-05 on Esams host cp3052. Once the cache populated again, we can see that response start performance goes back to its pre-upgrade levels: {F32377033} Which isn't the case for the rest of Esams hosts, which were not rolled back and whose response start stayed elevated: {F32377067} {F32377089} {F32377090} {F32377091} {F32377092} {F32377093} {F32377094}

We're seeing [[ https://grafana.wikimedia.org/d/000000143/navigation-timing?orgId=1&var-source=navtiming2&var-metric=responseStart&var-percentile=All&from=now-30d&to=now | a spike of latency across the board ]] from real users' Navigation Timing responseStart metric that seems to start around 2020-09-28After seeing strong time-correlation between the response start regression and the rollout of Varnish 6 per DC for Eqsin on 2020-09-25: {F32370851}7027} Is there ongoing maintenanceAnd for Esams on 2020-09-29: {F32377025} Varnish was rolled back to its previous version on 2020-10-05 on Esams host cp3052. Once the cache populated again, we can see that response start performance goes back to its pre-upgrade levels: {F32377033} Which isn't the case for the rest of Esams hosts, outage or work from #traffic that could explain this?which were not rolled back and whose response start stayed elevated: {F32377067} {F32377089} {F32377090} {F32377091} {F32377092} {F32377093} {F32377094}