It appears that there is a performance regression for #GrowthExperiments Special:Homepage feature coinciding with the rollout of wmf.28 ([Grafana](https://grafana.wikimedia.org/d/vGq7hbnMz/special-homepage-and-suggested-edits?orgId=1)).
Last 7 days (compared to 7 days prior in orange):
30 day view:
Note that there are several samples recording ~2 seconds to load Special:Homepage on desktop, compared to ~700ms which we usually expect.
The spike ([logs for the day](https://sal.toolforge.org/production?p=8&q=&d=2022-09-06)) coincides with:
* 2022-09-06 03:02 `Started scap: testwikis wikis to 1.39.0-wmf.28 refs T314189` [log entry](https://sal.toolforge.org/log/swy_EIMBa_6PSCT92R7j)
* 2022-09-06 03:26 `multi-DC stage 4: all traffic to appservers-ro, rolling out via puppet 03:24-03:54` ([log entry](https://sal.toolforge.org/log/UJHVEIMB8Fs0LHO5hI1f))
At first glance, I don't see any changes in GrowthExperiments or MediaWiki core that would account for the performance regression.
Candidates for contributors to this:
- ~~image-recommendation cache fetch~~ see T317187#8220110
- ~~wmf.27-wmf.28 changes~~ – unlikely given that we observe the spikes when the train is at group0 and most of traffic to Special:Homepage comes from group1/group2 wikis
- ElasticSearch config switch patch, deployed prior to wmf.28 rollout.
- multi-DC deployment
- php7.4 deployment