Page MenuHomePhabricator

Reportupdater outputs files with restricted permissions
Closed, ResolvedPublic4 Estimated Story Points


Right now any file in /srv/published-datasets/discovery modified by discovery-stats account on stat1005 (via statistics::discovery) has permissions mode 0644 which prevents Chelsy and me from modifying them (sometimes we need to, like right now).

It should be 0664 so that members of the group could modify those datasets.

Event Timeline

Change 371955 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[analytics/reportupdater@master] Give group write permission to output files

elukey lowered the priority of this task from Unbreak Now! to Medium.Aug 18 2017, 1:01 PM
elukey added a subscriber: elukey.

Added a couple of notes on IRC:

14:55  <elukey> I am wondering where are the discovery reportupdater jobs in puppet
14:57  <elukey> ah /srv/discovery/golden/
14:59  <elukey> bearloga: o/ - If you guys start all the report updater stuff via the above script, you might just add an
                appropriate umask to the script and this should do the trick
14:59  <elukey> discovery-stats@stat1005:/srv/published-datasets$ umask -S
14:59  <elukey> u=rwx,g=rx,o=rx
14:59  <elukey> so by default files are created with rx perms for the target group
14:59  <elukey> I would use umask rather than hardcoding perms to report updater

Hi all!

There is one gziped log that has +r for all users: /srv/discovery/log/golden-daily.log-20170806.gz. Looking at it, you can see that for the last days has been failing all the time. The most common errors are variations of:

2017-08-06 05:37:02,480 - ERROR - Report "clickthrough_breakdown" could not be written because of error: Could not open the temporary output file ([Errno 13] Permission denied: '/srv/published-datasets/discovery/metrics/portal/clickthrough_breakdown.tsv.tmp').

I have seen that some of the report directories in /srv/published-datasets/discovery belong to the bearloga user. Is it possible that after Aug 14 some directories have been changed to that user? This would explain the permission failures.

Regarding reports with empty rows: The only way outputs empty fields (resulting in empty rows) is when the query/script returns no rows/no results (or null results). This might mean that maybe an R script is also having permission problems. I don't know though if they also read/write files. I guess the forecasting scripts read previous data?

Yea, and I totally +1 @elukey 's suggestion to use umask. This should IMO solve all problems: the inaccessible log files, the update_reports.p permission errors, the permission errors, and the rows with empty fields. If is used by hand in the future, I think it should be executed also with umask or sudo -u discovery-stats.

As main autor of reportupdater, I totally agree with you on that reportupdater should not create files in source directories. Created this task to solve that: T173604. Thanks!

Mentioned in SAL (#wikimedia-operations) [2017-08-23T18:40:28Z] <gehel> resetting permissions on stat1005:/srv/published-datasets/discovery - T173333

Mentioned in SAL (#wikimedia-operations) [2017-08-23T18:42:59Z] <gehel> manually running report updater for discovery golden data on stat1005 - T173333

Change 373506 had a related patch set uploaded (by Gehel; owner: Gehel):
[wikimedia/discovery/golden@master] Set umask to ensure files are created with correct permissions.

Mentioned in SAL (#wikimedia-operations) [2017-08-24T12:45:39Z] <gehel> killing discovery report updater on stats1005 (stuck since Aug 15) - T173333

Change 373506 merged by Bearloga:
[wikimedia/discovery/golden@master] Set umask to ensure files are created with correct permissions.

Change 371955 abandoned by Bearloga:
Give group write permission to output files

Abandoning in favor of the umask solution. Thanks everyone!

mpopov moved this task from Next Up to Done on the Analytics-Kanban board.

This particular issue appears to be fixed. Will reopen if we find out otherwise :) Thanks @elukey @mforns @Gehel! *thumbs up*