varnish-be not restarting correctly because of disk space issues
Closed, ResolvedPublic
Actions

Description

On cp3040 /etc/cron.d/varnish-backend-restart failed to properly restart varnish-be. From the logs:

Nov 03 08:20:38 cp3040 varnishd[7503]: WARNING: (-spersistent) file size does not fit in reported free space, continuing anyways!

After that, the following message got reported a few times:

Error: (-spersistent): fallocate() for file /srv/sda3/varnish.main1 failed: No space left on device

Restarting the service by hand with service varnish restart solved the issue.

It's worth noting that /usr/local/sbin/varnish-backend-restart stops varnish-be, wipes /srv/sd*/varnish* and then starts varnish-be again.

Related Gerrit Patches:

operations/puppet : production

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 319596 had a related patch set uploaded (by Ema):
varnish-backend-restart: workaround fallocate issues

ema triaged this task as Medium priority.Nov 3 2016, 4:12 PM

Change 319596 merged by BBlack:
varnish-backend-restart: workaround fallocate issues

Mentioned in SAL (#wikimedia-operations) [2016-11-07T11:40:39Z] <ema> cp3043: repool varnish-be and varnish-be-rand (T149881)

This was always varnish3 instances with persistent storage AFAIK, and those are gone now.