Page MenuHomePhabricator

varnish-be not restarting correctly because of disk space issues
Closed, ResolvedPublic

Description

On cp3040 /etc/cron.d/varnish-backend-restart failed to properly restart varnish-be. From the logs:

Nov 03 08:20:38 cp3040 varnishd[7503]: WARNING: (-spersistent) file size does not fit in reported free space, continuing anyways!

After that, the following message got reported a few times:

Error: (-spersistent): fallocate() for file /srv/sda3/varnish.main1 failed: No space left on device

Restarting the service by hand with service varnish restart solved the issue.

It's worth noting that /usr/local/sbin/varnish-backend-restart stops varnish-be, wipes /srv/sd*/varnish* and then starts varnish-be again.

Details

Related Gerrit Patches:

Event Timeline

ema created this task.Nov 3 2016, 8:39 AM
Restricted Application added a project: Operations. · View Herald TranscriptNov 3 2016, 8:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 319596 had a related patch set uploaded (by Ema):
varnish-backend-restart: workaround fallocate issues

https://gerrit.wikimedia.org/r/319596

ema triaged this task as Medium priority.Nov 3 2016, 4:12 PM

Change 319596 merged by BBlack:
varnish-backend-restart: workaround fallocate issues

https://gerrit.wikimedia.org/r/319596

Mentioned in SAL (#wikimedia-operations) [2016-11-07T11:40:39Z] <ema> cp3043: repool varnish-be and varnish-be-rand (T149881)

ema moved this task from Triage to Caching on the Traffic board.Nov 7 2016, 11:41 AM
BBlack closed this task as Resolved.Nov 15 2016, 1:52 PM
BBlack claimed this task.

This was always varnish3 instances with persistent storage AFAIK, and those are gone now.