varnish crash upon reload after libvmod-netmapper upgrade due to liburcu6 assertion
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ema
	Oct 28 2020, 10:38 AM

Description

After upgrading libvmod-netmapper on cp4028 to tackle T266567, I've reloaded the varnish-frontend instance to ensure it wouldn't crash. It did crash due to the following liburcu6 assertion:

Oct 28 10:26:39 cp4028 varnishd[20043]: Child (49027) said varnishd: urcu-qsbr.c:470: rcu_register_thread_qsbr: Assertion `!URCU_TLS(rcu_reader).registered' failed.
Oct 28 10:26:40 cp4028 varnishd[20043]: Child (49027) died signal=6
Oct 28 10:26:40 cp4028 varnishd[20043]: Child (49027) Panic at: Wed, 28 Oct 2020 10:26:40 GMT
                                        Wrong turn at cache/cache_main.c:284:
                                        Signal 6 (Aborted) received at 0x700000bf83 si_code -6
                                        version = varnish-6.0.6 revision 29a1a8243dbef3d973aec28dc90403188c1dc8e7, vrt api = 7.1

Luckily I tried the reload on one host before upgrading all other libvmod-netmappers, or multiple instances would have crashed upon the first pool/depool. Canceling the fleet-wide libvmod-netmapper upgrade.

Related Objects

Mentioned In: T264398: 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1)
T266567: libvmod-netmapper: must specify ABI stanza
Mentioned Here: T264398: 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1)
T266567: libvmod-netmapper: must specify ABI stanza

Event Timeline

• ema created this task.Oct 28 2020, 10:38 AM

Restricted Application added a project: SRE. · View Herald TranscriptOct 28 2020, 10:38 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• ema triaged this task as High priority.Oct 28 2020, 10:38 AM

Mentioned in SAL (#wikimedia-operations) [2020-10-28T10:39:33Z] <ema> due to T266651, cancel the entry above: A:cp upgrade libvmod-netmapper to 1.9-1 T266567 T264398

Stashbot mentioned this in T266567: libvmod-netmapper: must specify ABI stanza.Oct 28 2020, 10:39 AM

Stashbot mentioned this in T264398: 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1).

• ema moved this task from Backlog to Bug Reports on the Traffic board.Oct 30 2020, 9:59 AM

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

BBlack moved this task from Backlog to Close or Untag? on the Traffic-Icebox board.Apr 13 2022, 8:04 PM

@Vgutierrez, since you were involved with the libvmod-netmapper upgrades, would you say that this 2-year-old issue is fixed?

Vgutierrez closed this task as Resolved.Jan 10 2023, 9:57 AM

Vgutierrez claimed this task.

varnish crash upon reload after libvmod-netmapper upgrade due to liburcu6 assertionClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

varnish crash upon reload after libvmod-netmapper upgrade due to liburcu6 assertion
Closed, ResolvedPublic
Actions