Page MenuHomePhabricator

varnish crash upon reload after libvmod-netmapper upgrade due to liburcu6 assertion
Open, HighPublic

Description

After upgrading libvmod-netmapper on cp4028 to tackle T266567, I've reloaded the varnish-frontend instance to ensure it wouldn't crash. It did crash due to the following liburcu6 assertion:

Oct 28 10:26:39 cp4028 varnishd[20043]: Child (49027) said varnishd: urcu-qsbr.c:470: rcu_register_thread_qsbr: Assertion `!URCU_TLS(rcu_reader).registered' failed.
Oct 28 10:26:40 cp4028 varnishd[20043]: Child (49027) died signal=6
Oct 28 10:26:40 cp4028 varnishd[20043]: Child (49027) Panic at: Wed, 28 Oct 2020 10:26:40 GMT
                                        Wrong turn at cache/cache_main.c:284:
                                        Signal 6 (Aborted) received at 0x700000bf83 si_code -6
                                        version = varnish-6.0.6 revision 29a1a8243dbef3d973aec28dc90403188c1dc8e7, vrt api = 7.1

Luckily I tried the reload on one host before upgrading all other libvmod-netmappers, or multiple instances would have crashed upon the first pool/depool. Canceling the fleet-wide libvmod-netmapper upgrade.

Event Timeline

ema created this task.Oct 28 2020, 10:38 AM
Restricted Application added a project: Operations. · View Herald TranscriptOct 28 2020, 10:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as High priority.Oct 28 2020, 10:38 AM

Mentioned in SAL (#wikimedia-operations) [2020-10-28T10:39:33Z] <ema> due to T266651, cancel the entry above: A:cp upgrade libvmod-netmapper to 1.9-1 T266567 T264398

ema moved this task from Triage to Bug Reports on the Traffic board.Oct 30 2020, 9:59 AM