Page MenuHomePhabricator

Report updater setting log ownership incorrectly (leading to cronspam)
Closed, ResolvedPublic3 Estimated Story Points

Description

Sometimes I get the following email from stat1005, sent to root@ due to logrotate failures:

/etc/cron.daily/logrotate:
error: error setting owner of /srv/reportupdater/log/limn-language-data-interlanguage.log-20180401.gz to uid 116 and gid 122: Operation not permitted
run-parts: /etc/cron.daily/logrotate exited with return code 1

In this case:

elukey@stat1005:~$ ls -l  /srv/reportupdater/log/limn-language-data-interlanguage.log-20180401.gz
-rw------- 1 hdfs wikidev 0 Apr  9 06:25 /srv/reportupdater/log/limn-language-data-interlanguage.log-20180401.gz

Complete view:

elukey@stat1005:~$ ls -l  /srv/reportupdater/log/limn-language-data-interlanguage.log*
-rw------- 1 hdfs wikidev  48976 Apr 10 06:00 /srv/reportupdater/log/limn-language-data-interlanguage.log
-rw-r--r-- 1 hdfs hdfs    348610 Apr  1 06:00 /srv/reportupdater/log/limn-language-data-interlanguage.log-20180401
-rw------- 1 hdfs wikidev      0 Apr  9 06:25 /srv/reportupdater/log/limn-language-data-interlanguage.log-20180401.gz

Logrotate settings:

elukey@stat1005:~$ cat /etc/logrotate.d/reportupdater
# This file is managed by Puppet.

/srv/reportupdater/log/*.log {
       notifempty
       maxage 30
       rotate 2
       dateext
       compress
       delaycompress
       missingok
       su hdfs wikidev
}

In theory, we have the following on stat1005:

elukey@stat1005:~$ cat /etc/profile.d/umask-wikidev.sh
# !this file is managed by puppet!
# set umask to 0002 for wikidev users
# to prevent broken repos per T79400
if groups | grep -w -q wikidev; then
  umask 0002
else
  umask 0022
fi

So if a user is in wikidev it inherits the 0002 umask (so user/group allowed to write, not only user). The hdfs user is not in this group, so the umask is not set as we wished.

Event Timeline

fdans triaged this task as Medium priority.Apr 12 2018, 4:54 PM
fdans moved this task from Incoming to Operational Excellence on the Analytics board.

Change 428561 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] reportupdater: create logfile after rotation

https://gerrit.wikimedia.org/r/428561

Change 428561 merged by Elukey:
[operations/puppet@production] reportupdater: create logfile after rotation

https://gerrit.wikimedia.org/r/428561

This change should always create a file with 644 perms and correct username/group after every rotation. Keeping it open for a couple of days to check if it works as expected.

elukey edited projects, added Analytics-Kanban; removed Patch-For-Review.
elukey moved this task from Next Up to Done on the Analytics-Kanban board.
elukey set the point value for this task to 3.