Page MenuHomePhabricator

cp3059 Varnish child crash: Worker Pool Queue does not move
Closed, ResolvedPublic

Description

We had a varnish crash on cp3059:

Jul 01 14:07:43 cp3059 varnishd[1204]: Child (1694) Panic at: Thu, 01 Jul 2021 14:07:43 GMT
                                       Wrong turn at cache/cache_wrk.c:540:
                                       Worker Pool Queue does not move
                                       version = varnish-6.0.7 revision 525d371e3ea0e0c38edd7baf0f80dc226560f26e, vrt api = 7.1
                                       ident = Linux,4.19.0-14-amd64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll
                                       now = 12110617.922957 (mono), 1625148433.038834 (real)
                                       Backtrace:
                                         0x55e7ad786ccd: /usr/sbin/varnishd(+0x4dccd) [0x55e7ad786ccd]
                                         0x55e7ad7f1153: /usr/sbin/varnishd(VAS_Fail+0x13) [0x55e7ad7f1153]
                                         0x55e7ad7a8a0f: /usr/sbin/varnishd(pool_herder+0x7bf) [0x55e7ad7a8a0f]
                                         0x7f5ef5de6fa3: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f5ef5de6fa3]
                                         0x7f5ef5d174cf: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f5ef5d174cf]
                                       thread = (pool_herder)
                                       thr.req = (nil) {
                                       },
                                       thr.busyobj = (nil) {
                                       },
                                       vmods = {
                                         std = {Varnish 6.0.7 525d371e3ea0e0c38edd7baf0f80dc226560f26e, 0.0},
                                         header = {Varnish 6.0.6 29a1a8243dbef3d973aec28dc90403188c1dc8e7, 7.1},
                                         directors = {Varnish 6.0.7 525d371e3ea0e0c38edd7baf0f80dc226560f26e, 0.0},
                                         netmapper = {Varnish 6.0.6 29a1a8243dbef3d973aec28dc90403188c1dc8e7, 7.1},
                                         vsthrottle = {Varnish 6.0.6 29a1a8243dbef3d973aec28dc90403188c1dc8e7, 7.1},
                                       },

The host is running 6.0.7-1wm1 like the rest of the fleet.

Event Timeline

ema triaged this task as Medium priority.Jul 1 2021, 2:44 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-01T16:11:54Z] <vgutierrez> restart varnish-fe on cp3059 - T285953

BBlack claimed this task.
BBlack added a subscriber: BBlack.

We have a new varnish version coming soon, so stale crash reports are probably of little value now.