Change Details

On Thu 16 Dec 2021, the train for 1.38.0-wmf.13 rolled out to all wikis (refs T293954). Two days later, a Grafana alert started firing for insufficient "ResourceLoader: INM Satisfaction". I missed this as the time, and it has been firing on and off since then. The INM Satisfaction ratio is the % of "If-None-Match requests" from end-user web browsers that our CDN is able to respond to with a "304 Not Modified" response. After 2 days of churn following a train deploy, this is usually well above 80%. <https://grafana.wikimedia.org/d/000000402/resourceloader-alerts?orgId=1&from=1638316800000&to=1641167999000> {F34907080} This last train, it never went above 70%, despite there being no other train deployments after it for almost a month. We know that browser's http cache rarely lasts longer than two weeks, so it makes no sense that it remains stuck at 70% satisfaction. The alert panel has a useful info text: > If this ratio is low, it may indicate a problem with cache churn on Varnish or HTTP clients. As of writing (Mar 2019) it is typically well above 80%. > > After a deployment that affects most traffic (eg. MW branch to group2, or major backport), then it is normal to drop to below 30%. This is expected. It should take less than 2 days to reach 80%. As such, I suppose it is likely then that the wmf.13 train included a change that caused a module to have an unstable version hash.

On Thu 16 Dec 2021, the train for 1.38.0-wmf.13 rolled out to all wikis (refs T293954). Two days later, a Grafana alert started firing for insufficient "ResourceLoader: INM Satisfaction". I missed this as the time, and it has been firing on and off since then. The INM Satisfaction ratio is the % of "If-None-Match requests" from end-user web browsers that our CDN is able to respond to with a "304 Not Modified" response. After 2 days of churn following a train deploy, this is usually well above 80%. <https://grafana.wikimedia.org/d/000000402/resourceloader-alerts?orgId=1&from=1638316800000&to=1641167999000> {F34908534} This last train, it never went above 70%, despite there being no other train deployments after it for almost a month. We know that browser's http cache rarely lasts longer than two weeks, so it makes no sense that it remains stuck at 70% satisfaction. The alert panel has a useful info text: > If this ratio is low, it may indicate a problem with cache churn on Varnish or HTTP clients. As of writing (Mar 2019) it is typically well above 80%. > > After a deployment that affects most traffic (eg. MW branch to group2, or major backport), then it is normal to drop to below 30%. This is expected. It should take less than 2 days to reach 80%. As such, I suppose it is likely then that the wmf.13 train included a change that caused a module to have an unstable version hash.

On Thu 16 Dec 2021, the train for 1.38.0-wmf.13 rolled out to all wikis (refs T293954). Two days later, a Grafana alert started firing for insufficient "ResourceLoader: INM Satisfaction". I missed this as the time, and it has been firing on and off since then. The INM Satisfaction ratio is the % of "If-None-Match requests" from end-user web browsers that our CDN is able to respond to with a "304 Not Modified" response. After 2 days of churn following a train deploy, this is usually well above 80%. <https://grafana.wikimedia.org/d/000000402/resourceloader-alerts?orgId=1&from=1638316800000&to=1641167999000> {F34907080}8534} This last train, it never went above 70%, despite there being no other train deployments after it for almost a month. We know that browser's http cache rarely lasts longer than two weeks, so it makes no sense that it remains stuck at 70% satisfaction. The alert panel has a useful info text: > If this ratio is low, it may indicate a problem with cache churn on Varnish or HTTP clients. As of writing (Mar 2019) it is typically well above 80%. > > After a deployment that affects most traffic (eg. MW branch to group2, or major backport), then it is normal to drop to below 30%. This is expected. It should take less than 2 days to reach 80%. As such, I suppose it is likely then that the wmf.13 train included a change that caused a module to have an unstable version hash.