Wed, May 15
Please provide the full responses, including headers, returned by the HHVM and PHP7 origin servers.
Actually no, we did fix the issue at the Swift layer (T162348), hence we removed the workaround from Varnish: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/348699/. That means that there is nothing wrong with ATS.
The issue is indeed reproducible again, affecting ATS hosts.
Mon, May 13
Fri, May 10
Thu, May 9
Tue, May 7
IPMI seems to be working remotely:
All Varnish backends in ulsfo upload replaced with ATS.
I've ack'ed the warnings in Icinga for the time being.
Interestingly, there was a memory usage spike right before the host crashed.
Mon, May 6
Fri, May 3
Thu, May 2
This is now fixed, CL matches the actual body length:
Tue, Apr 30
Mon, Apr 29
VTC tests can now be run from dev workstations against PCC:
Sat, Apr 27
Fri, Apr 26
Wed, Apr 24
Indeed our Varnish mailbox lag Icinga check only applies to Varnish backends, given that backends are those affected by T145661 and similar issues. During the Puppet refactoring splitting frontend/backend puppetization (T219967) I forgot to move the check from the Varnish module, where it shouldn't have been in the first place, to the backend profile. Doing this will ensure that the check is only added to cache hosts using Varnish as the cache backend software, not those using ATS such as cp4021.
Apr 23 2019
Fixed the former, deleted the latter. Thanks for the reminder!
Apr 18 2019
Apr 17 2019
Apr 16 2019
@Ottomata thanks! The new error message is helpful, and the proposed solution works.
Switchback completed, closing.
Apr 11 2019
Apr 10 2019
Apr 9 2019
Apr 8 2019
Apr 5 2019
Apr 4 2019
Here is how our custom ATS errors look like.