When executing LibreNMS (sudo -u librenms php /srv/deployment/librenms/librenms/poller.php -h 2 -d) the following error appears:
Warning: fopen(/var/log/librenms.log): failed to open stream: Permission denied in /srv/deployment/librenms/librenms-cache/revs/de2bd0369fc46effba4a4ca9ebafc95b40b1af22/includes/functions.php on line 150.
This happens because the /var/log folder belongs to the root user therefore the librenms user can't create the folder due to insufficient permissions.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | andrea.denisse | T309074 Put netmon1003 in service | |||
| Resolved | BUG REPORT | andrea.denisse | T315393 Logrotate is unable to rotate LibreNMS logs in the netmon instances due to insuficient permissions to read and write log files in /var/log/ |
Change 823764 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):
[operations/puppet@production] netmon: Create LibreNMS logs file.
The LibreNMS logrotate configuration should create the /var/log/librenms.log file which it's not creating.
To ensure that LibreNMS can write logs and Lograte con rotate them the librenms.log is explicitly created using Puppet, the logs are now being stored inside the /var/log/librenms/librenms.log folder which LibreNMS and Logrotate have access to and the Logrotate and LibreNMS configurations were updated to point to the correct file.
This changes are added in patch #823764.
Change 823764 merged by Andrea Denisse:
[operations/puppet@production] netmon: Configure Logrotate for LibreNMS logs
Seeing this today:
Aug 25 00:00:01 netmon1003 logrotate[3061772]: error: skipping "/var/log/librenms/librenms.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
Change 826771 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):
[operations/puppet@production] netmon: Configure logrotate to rotate logs as the 'librenms' user.
Thanks for reporting the issue @herron .
The /var/log/librenms directory has the following permissions: drwxrwxr-x www-data librenms.
It's not world writable but it does belong to a group that is not the 'root' group, it belongs to the 'librenms' group.
Adding the 'su' directive to rotate the 'librenms' logs as the 'librenms' user fixes the issue. Sent in patch #826771.
Validated using sudo logrotate --debug /etc/logrotate.d/librenms with the new config.
Change 826771 merged by Andrea Denisse:
[operations/puppet@production] netmon: Configure logrotate to rotate logs as the 'librenms' user.
Note this solved the alert on netmon1003 but not on 1002 or 2001:
logrotate[10610]: error: error setting owner of /var/log/librenms/librenms.log to uid 496 and gid 0
Not sure if this is a known issue with UIDs, or permissions have to be enforced on puppet, or something else?
I would expect that issue to be gone once https://gerrit.wikimedia.org/r/c/operations/puppet/+/826431/ gets merged.
I'm still working on this issue, it's tough because the /var/log/librenms/librenms.log.1.gz file has the following permissions: -rw------- 1 librenms librenms and logrotate wants to make the log rotation as the user with UID 33, that is the 'www-data' user even though in the logrotate configuration it's specified to perform log rotation as the 'librenms' user.
I guess it may be inheriting the UID from the parent directory (/var/log/librenms) as that directory belongs to the www-data user.
An interesting thing to note is that this issue is only present in the 'netmon1002' and 'netmon2001' instances, both of them use Debian Buster so it's possibly an issue with Debian Buster logrotate version.
The issue is not present in 'netmon1003' that uses Debian Bullseye.
Digging deeper into the issue I found that having fleetwide uid and gid mappings for the Netmon instances did not solve the issue.
I noticed that the librenms.log file is owned by librenms:librenms as specified in the librenms Puppet manifest. The librenms:librenms user/group is used for rotation as specified in the logrotate configuration however, rotated logs were owned by the www-data:librenms user/group and not by the librenms:librenms user/group.
Looking at the logrotate source code I noticed that the createOutputFile function uses the O_CREAT flag which may explain why the rotated logs have a different owner than the one specified in the logrotate configuration. This is a little bit confusing as I'd expect the rotated files to belong to the librenms:librenms user/group as that's the user/group used for the rotation but they seem to be inheriting the user/group from either the containing directory or from the Apache2 process itself as that's the process that writes the logs.
An important thing to note is that the logs are written by the www-data user, I assume this happens because that's the user running the Apache2 process therefore, the permissions for the /var/log/librenms (www-data:librenms) directory can be changed from 775 to 755 as the group librenms does not need write permissions in that directory.
Modifying the logrotate configuration to set the expected user/group solves the issue.
Change 827450 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):
[operations/puppet@production] netmon: Rotate logs as the www-data user and librenms group.
Change 827450 merged by Andrea Denisse:
[operations/puppet@production] netmon: Rotate logs as the www-data user and librenms group.