Page MenuHomePhabricator

Ensure that the hive metastore logging on analytics coordinator hosts is correctly configured
Open, MediumPublic

Description

We recently observed a disk full event on an-coord1003

The largest consumer of disk space was the /var/log/hive directory.

image.png (896×1 px, 435 KB)

It appears that we had at least 5 months' worth of hive-metastore.log.YY-MM-DD files in this directory.

There is a log4j configuration file at: /etc/hive/conf/hive-log4j2.properties

This seems to suggest that we would be keeping 14 days' worth of log files, but perhaps this isn't being applied properly.

We should ensure that we have suitable housekeeping in place for these hive metastore logs.