Page MenuHomePhabricator

HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk
Closed, InvalidPublic


deployment-mediawiki02 labs instance only has 2GB of disk which highlight an hhvm issue. The hhvm upstart process emits log to /var/log/upstart/hhvm.log which contains a bunch of php notices. That is also send to syslog ( /var/log/syslog/ ) creating a snowball effect that completely fill up /var/.

The hhvm upstart job should be made to discard anything that is below the 'error' level and the notices be sent to a central syslog.

Version: unspecified
Severity: major



Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:40 AM
bzimport set Reference to bz69976.
bzimport added a subscriber: Unknown Object (MLST).

changed importance to "major", this makes beta hard to use for real people and causes a lot of false test failures.

Bryan told me the log issue is apparently fixed in production. Since mediawiki02 does not run puppet, it is not taking in account the recent changes made in operations/puppet.git :-(

I think puppet is now passing on the hhvm instances. I am not sure where the log are written to though.

yuvipanda claimed this task.

Haven't seen a full disk error related to logs on mediawiki* machines in a while...

Reopening, the reason I filled this bug is hhvm emitting lot of message on stdout/stderr which are caught by upstart and fill both:

  • /var/log/upstart/hhvm.log
  • /var/log/syslog

The small /var on labs instances exacerbate the disk consumption which I believe need to be constrained / fixed since that is surely going to hit us in production as well.

Will rephrase summary.

Only way to increase /var is to re-create instances. New instances have a re-sizable /var

hashar renamed this task from deployment-mediawiki02 labs instance only has 2GB of disk to HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk.Nov 24 2014, 2:19 PM
hashar removed yuvipanda as the assignee of this task.
hashar updated the task description. (Show Details)
hashar added projects: HHVM, acl*sre-team.
hashar set Security to None.
hashar added subscribers: yuvipanda, Joe.
greg triaged this task as Medium priority.Nov 24 2014, 11:33 PM
greg moved this task from To Triage to Next: Maintenance on the Beta-Cluster-Infrastructure board.

Just FTR, this is solved and the title of the bug is misleading. Resolving.