Page MenuHomePhabricator
Create Task
Maniphest T224694

cp3041 - Varnish frontend child restarted icinga alert
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

See: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=cp3041&service=Varnish+frontend+child+restarted

2 gt 1

Also that alert doesn't have a runbook, might be worth adding one.

By (I think) adding a note_url field, similar to: https://github.com/wikimedia/puppet/commit/12093fb9c98873e04c790109bdb1e606694acb0f

Details

SubjectRepoBranchLines +/-
cache: stop passing gethdr_extrachance to varnishoperations/puppetproduction+0 -23
cp4027: upgrade Varnish to 5.1.3-1wm10operations/puppetproduction+7 -0
cp1075: do not pass gethdr_extrachance=0operations/puppetproduction+7 -0
Add 0020-assert-error-http1_minimal_response.patchoperations/debs/varnish4debian-wmf+24 -0
Customize query in gerrit

Related Objects

Event Timeline

• ayounsi triaged this task as Medium priority.May 30 2019, 7:12 PM
• ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptMay 30 2019, 7:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
BBlack added a comment.May 30 2019, 7:43 PM

That alert basically means that a varnish frontend daemon crashed (and as usual was auto-restarted by a manager process). These are pretty rare and usually worth some investigation.

Runbook should probably say to gather up the crash info from syslog and attach it to a private paste or task for later analysis (this isn't just about user PII - the tech details of the crash could highlight a potential exploitable bug we want to avoid drawing attention to until we understand and/or mitigate).

I think the only way to clear the counter it is to do a full frontend restart, or we can just ACK it with a task# and leave it be.

Private paste for this one: P8576

gerritbot added a comment.Jun 3 2019, 11:24 AM

Change 513977 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/varnish4@debian-wmf] Add 0020-assert-error-http1_minimal_response.patch

https://gerrit.wikimedia.org/r/513977

gerritbot added a project: Patch-For-Review.Jun 3 2019, 11:24 AM
• ema moved this task from Backlog to Caching on the Traffic board.Jun 3 2019, 3:09 PM
gerritbot added a comment.Jun 6 2019, 8:23 AM

Change 513977 merged by Ema:
[operations/debs/varnish4@debian-wmf] Add 0020-assert-error-http1_minimal_response.patch

https://gerrit.wikimedia.org/r/513977

Stashbot added a comment.Jun 6 2019, 10:30 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-06T10:30:01Z] <ema> varnish 5.1.3-1wm10 uploaded to stretch-wikimedia T224694

gerritbot added a comment.Jun 6 2019, 11:52 AM

Change 514699 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp1075: stop passing gethdr_extrachance=0

https://gerrit.wikimedia.org/r/514699

gerritbot added a comment.Jun 6 2019, 11:58 AM

Change 514699 merged by Ema:
[operations/puppet@production] cp1075: do not pass gethdr_extrachance=0

https://gerrit.wikimedia.org/r/514699

Stashbot added a comment.Jun 6 2019, 12:00 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-06T12:00:29Z] <ema> cp1075: upgrade varnish to 5.1.3-1wm10 T224694

Stashbot added a comment.Jun 6 2019, 12:11 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-06T12:11:04Z] <ema> cp1075: repool with varnish 5.1.3-1wm10 T224694

gerritbot added a comment.Jun 17 2019, 5:18 AM

Change 517359 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: stop passing gethdr_extrachance to varnish

https://gerrit.wikimedia.org/r/517359

gerritbot added a comment.Jun 17 2019, 1:06 PM

Change 517415 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp4027: upgrade Varnish to 5.1.3-1wm10

https://gerrit.wikimedia.org/r/517415

gerritbot added a comment.Jun 17 2019, 1:19 PM

Change 517415 merged by Ema:
[operations/puppet@production] cp4027: upgrade Varnish to 5.1.3-1wm10

https://gerrit.wikimedia.org/r/517415

Stashbot added a comment.Jun 17 2019, 1:25 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-17T13:25:11Z] <ema> cp4027: upgrade Varnish packages to 5.1.3-1wm10 T224694

Stashbot added a comment.Jun 17 2019, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-17T13:45:22Z] <ema> reboot cp4027 for dist and Varnish upgrade T224694

gerritbot added a comment.Jun 18 2019, 9:41 AM

Change 517359 merged by Ema:
[operations/puppet@production] cache: stop passing gethdr_extrachance to varnish

https://gerrit.wikimedia.org/r/517359

Stashbot added a comment.Jun 18 2019, 12:27 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-18T12:27:34Z] <ema> cp5007 (varnish-be text): reboot for kernel and varnish upgrade T224694

Stashbot added a comment.Jun 18 2019, 12:49 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-18T12:49:09Z] <ema> cp3034 (ats-be upload) cp2002 (varnish-be upload): reboot for kernel and varnish upgrade T224694

Stashbot added a comment.Jun 18 2019, 1:10 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-18T13:10:04Z] <ema> cache nodes: begin rolling reboots for kernel and varnish upgrades T224694

Stashbot added a comment.Jun 18 2019, 4:07 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-18T16:07:38Z] <ema> cache nodes: stop rolling reboots for today, 17/80 done T224694

Stashbot added a comment.Jun 19 2019, 8:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-19T08:01:26Z] <ema> cache nodes: resume rolling reboots for kernel and varnish upgrades T224694

Stashbot added a comment.Jun 19 2019, 11:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-19T11:07:13Z] <ema> cache nodes: pause rolling reboots for kernel and varnish upgrades T224694 T225998

Stashbot mentioned this in T225998: Study performance impact of disabling TCP selective acknowledgments.Jun 19 2019, 11:07 AM
Stashbot added a comment.Jun 19 2019, 12:20 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-19T12:20:48Z] <ema> cache nodes: resume rolling reboots for kernel and varnish upgrades T224694 T225998

Stashbot added a comment.Jun 19 2019, 4:01 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-19T16:01:26Z] <ema> cache nodes: stop rolling reboots for today, 47/80 done T224694 T225998

Stashbot added a comment.Jun 20 2019, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-20T09:17:36Z] <ema> cache nodes: resume rolling reboots for kernel and varnish upgrades T224694 T225998 T226048

Stashbot mentioned this in T226048: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster).Jun 20 2019, 9:17 AM
• ema closed this task as Resolved.Jun 21 2019, 2:21 PM
• ema claimed this task.

All cache nodes are currently running Varnish 5.1.3-1wm10, which fixes this.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits