Page MenuHomePhabricator

varnishmtail panics on buster
Closed, ResolvedPublic

Description

varnishmtail is having a bad time on cp4032 and panics quite often:

Jan 24 11:45:15 cp4032 varnishmtail[15172]: panic: runtime error: index out of range
Jan 24 11:45:15 cp4032 varnishmtail[15172]: goroutine 43 [running]:
Jan 24 11:45:15 cp4032 varnishmtail[15172]: bytes.(*Buffer).WriteByte(0xc000298000, 0xc00063e073, 0x0, 0x0)
Jan 24 11:45:15 cp4032 varnishmtail[15172]:         /usr/lib/go-1.11/src/bytes/buffer.go:271 +0xa8
Jan 24 11:45:15 cp4032 varnishmtail[15172]: bytes.(*Buffer).WriteRune(0xc000298000, 0x73, 0xd42, 0x73, 0x1)
Jan 24 11:45:15 cp4032 varnishmtail[15172]:         /usr/lib/go-1.11/src/bytes/buffer.go:281 +0x14c
Jan 24 11:45:15 cp4032 varnishmtail[15172]: github.com/google/mtail/internal/tailer.(*File).Read(0xc000286480, 0xc0002e1dd8, 0x1)
Jan 24 11:45:15 cp4032 varnishmtail[15172]:         github.com/google/mtail/internal/tailer/file.go:197 +0xf9
Jan 24 11:45:15 cp4032 varnishmtail[15172]: github.com/google/mtail/internal/tailer.(*File).Follow(0xc000286480, 0xc0001aae98, 0xc0001aae98)
Jan 24 11:45:15 cp4032 varnishmtail[15172]:         github.com/google/mtail/internal/tailer/file.go:143 +0x3b3
Jan 24 11:45:15 cp4032 varnishmtail[15172]: github.com/google/mtail/internal/tailer.doFollow(0xc000286480)
Jan 24 11:45:15 cp4032 varnishmtail[15172]:         github.com/google/mtail/internal/tailer/tail.go:203 +0x2f
Jan 24 11:45:15 cp4032 varnishmtail[15172]: github.com/google/mtail/internal/tailer.(*Tailer).handleLogEvent(0xc00027ff80, 0x7ffce3047f22, 0xa)
Jan 24 11:45:15 cp4032 varnishmtail[15172]:         github.com/google/mtail/internal/tailer/tail.go:198 +0x10a
Jan 24 11:45:15 cp4032 varnishmtail[15172]: github.com/google/mtail/internal/tailer.(*Tailer).run(0xc00027ff80, 0xc0001c69c0)
Jan 24 11:45:15 cp4032 varnishmtail[15172]:         github.com/google/mtail/internal/tailer/tail.go:293 +0x177
Jan 24 11:45:15 cp4032 varnishmtail[15172]: created by github.com/google/mtail/internal/tailer.New
Jan 24 11:45:15 cp4032 varnishmtail[15172]:         github.com/google/mtail/internal/tailer/tail.go:79 +0x193
Jan 24 11:45:15 cp4032 varnishmtail[15172]: Log overrun
Jan 24 11:45:15 cp4032 varnishmtail[15172]: Log reacquired
Jan 24 11:45:15 cp4032 systemd[1]: varnishmtail.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 24 11:45:15 cp4032 systemd[1]: varnishmtail.service: Failed with result 'exit-code'.

Event Timeline

Vgutierrez moved this task from Backlog to Caching on the Traffic board.

that panic comes from mtail itself, after all varnishmtail is running:

/usr/bin/varnishncsa -n frontend -c -b -F "${FMT}" | mtail -progs "${PROGS}" -logs /dev/stdin

on the same host, atsmtail looks happy, so it looks like it's something related to the varnish mtail progs

Mentioned in SAL (#wikimedia-operations) [2020-01-24T14:28:08Z] <vgutierrez> uploaded mtail 3.0.0~rc5-1~bpo9+1wmf2 to apt.wm.o (buster) - T243591

Vgutierrez claimed this task.

Solved after rebuilding mtail 3.0.0~rc5 for buster