Page MenuHomePhabricator
Create Task
Maniphest T249344

varnishd crashes in vbf_stp_condfetch(): cp3057 and cp3061
Closed, ResolvedPublic
Actions

Description

On April 2nd we had two varnish-fe crashes affecting cp3061 and cp3057:

Apr 02 12:13:42 cp3061 varnishd[29726]: Child (29751) Panic at: Thu, 02 Apr 2020 12:13:42 GMT
                                        Assert error in vbf_stp_condfetch(), cache/cache_fetch.c line 797:
                                          Condition((vbf_beresp2obj(bo)) == 0) not true.
Apr 02 12:12:36 cp3057 varnishd[14667]: Child (14721) Panic at: Thu, 02 Apr 2020 12:12:36 GMT
                                        Assert error in vbf_stp_condfetch(), cache/cache_fetch.c line 797:
                                          Condition((vbf_beresp2obj(bo)) == 0) not true.

I have reported the issue upstream together with a proposed patch: https://github.com/varnishcache/varnish-cache/issues/3273

Details

ProjectBranchLines +/-Subject
operations/debs/varnish4debian-wmf+109 -05.1.3-1wm13: handle fetch error in vbf_stp_condfetch
Customize query in gerrit

Related Objects

Event Timeline

• ema created this task.Apr 3 2020, 3:17 PM
Restricted Application added a project: SRE. · View Herald TranscriptApr 3 2020, 3:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ema triaged this task as Medium priority.Apr 3 2020, 3:17 PM
Stashbot added a comment.Apr 3 2020, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-03T15:18:31Z] <ema> cp3057: restart varnish-fe T249344

Stashbot added a comment.Apr 3 2020, 3:43 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-03T15:43:30Z] <ema> cp3061: restart varnish-fe T249344

• ema moved this task from Backlog to Caching on the Traffic board.Apr 6 2020, 7:11 AM
gerritbot added a comment.Apr 6 2020, 7:16 AM

Change 586299 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/varnish4@debian-wmf] 5.1.3-1wm13: handle fetch error in vbf_stp_condfetch

https://gerrit.wikimedia.org/r/586299

gerritbot added a project: Patch-For-Review.Apr 6 2020, 7:16 AM
gerritbot added a comment.Apr 6 2020, 8:57 AM

Change 586299 merged by Ema:
[operations/debs/varnish4@debian-wmf] 5.1.3-1wm13: handle fetch error in vbf_stp_condfetch

https://gerrit.wikimedia.org/r/586299

Stashbot added a comment.Apr 6 2020, 9:08 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-06T09:08:30Z] <ema> upload varnish 5.1.3-1wm13 to buster-wikimedia on apt1001.wm.org T249344

Stashbot added a comment.Apr 6 2020, 9:11 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-06T09:11:33Z] <ema> cp2027: upgrade varnish to 5.1.3-1wm13 and restart varnish-fe T249344

Stashbot added a comment.Apr 6 2020, 1:05 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-06T13:05:14Z] <ema> cache: upgrade varnish to 5.1.3-1wm13, begin rolling varnish-fe restarts T249344

• ema closed this task as Resolved.Apr 7 2020, 6:16 AM
• ema claimed this task.

5.1.3-1wm13 deployed.

Vgutierrez awarded a token.Apr 7 2020, 6:18 AM
• ema mentioned this in T249809: cache_upload varnish-fe exhausting transient memory.Apr 9 2020, 9:41 AM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL