Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
Revert "Increase nuke_limit in upload@eqsin" | operations/puppet | production | +1 -1 | |
Increase nuke_limit in upload@eqsin | operations/puppet | production | +1 -1 |
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Open | None | T274888 cp_upload @ eqsin cascading failures, February 2021 | |||
Resolved | CDanis | T275028 validate or revert the new large_objects_cutoff & nuke_limit settings on upload@eqsin |
Event Timeline
Change 664824 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] Increase nuke_limit in upload@eqsin
Change 664824 merged by CDanis:
[operations/puppet@production] Increase nuke_limit in upload@eqsin
Change 664657 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] Revert "Increase nuke_limit in upload@eqsin"
Change 664657 merged by CDanis:
[operations/puppet@production] Revert "Increase nuke_limit in upload@eqsin"
Mentioned in SAL (#wikimedia-operations) [2021-02-17T14:26:25Z] <cdanis> starting rolling restart of cp-upload@eqsin varnish-fe T275028
Mentioned in SAL (#wikimedia-operations) [2021-02-17T15:36:53Z] <cdanis> T275028 rolling restart done; check for fetch failures once caches re-fill
Caches have filled, and there's no more fetch failures due to "LRU limited": https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1&var-datasource=eqsin%20prometheus%2Fops&var-cache_type=upload&var-server=All&var-layer=frontend&from=1613576456135&to=1613678433179
So the new nuke_limit=1000 seems to be sufficient.