Right now any file in /srv/published-datasets/discovery modified by discovery-stats account on stat1005 (via statistics::discovery) has permissions mode 0644 which prevents Chelsy and me from modifying them (sometimes we need to, like right now).
It should be 0664 so that members of the group could modify those datasets.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T157807 Reinstall Analytics Hadoop Cluster with Debian Jessie | |||
| Declined | Ottomata | T159392 Update jq to v1.4.0 or higher | |||
| Resolved | Ottomata | T152712 Replacement of stat1002 and stat1003 | |||
| Resolved | Ottomata | T170471 Move statistics::discovery jobs from stat1002 -> stat1005 | |||
| Resolved | mpopov | T170494 [EPIC] Reconfigure Discovery-Stats on Analytics Cluster | |||
| Resolved | mforns | T173333 Reportupdater outputs files with restricted permissions |
Change 371955 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[analytics/reportupdater@master] Give group write permission to output files
Added a couple of notes on IRC:
14:55 <elukey> I am wondering where are the discovery reportupdater jobs in puppet
14:57 <elukey> ah /srv/discovery/golden/main.sh
14:59 <elukey> bearloga: o/ - If you guys start all the report updater stuff via the above script, you might just add an
appropriate umask to the script and this should do the trick
14:59 <elukey> discovery-stats@stat1005:/srv/published-datasets$ umask -S
14:59 <elukey> u=rwx,g=rx,o=rx
14:59 <elukey> so by default files are created with rx perms for the target group
14:59 <elukey> I would use umask rather than hardcoding perms to report updaterHi all!
There is one gziped log that has +r for all users: /srv/discovery/log/golden-daily.log-20170806.gz. Looking at it, you can see that for the last days update_reports.py has been failing all the time. The most common errors are variations of:
2017-08-06 05:37:02,480 - ERROR - Report "clickthrough_breakdown" could not be written because of error: Could not open the temporary output file ([Errno 13] Permission denied: '/srv/published-datasets/discovery/metrics/portal/clickthrough_breakdown.tsv.tmp').
I have seen that some of the report directories in /srv/published-datasets/discovery belong to the bearloga user. Is it possible that after Aug 14 some directories have been changed to that user? This would explain the update_reports.py permission failures.
Regarding reports with empty rows: The only way update_reports.py outputs empty fields (resulting in empty rows) is when the query/script returns no rows/no results (or null results). This might mean that maybe an R script is also having permission problems. I don't know though if they also read/write files. I guess the forecasting scripts read previous data?
Yea, and I totally +1 @elukey 's suggestion to use umask. This should IMO solve all problems: the inaccessible log files, the update_reports.p permission errors, the rerun_reports.py permission errors, and the rows with empty fields. If rerun_reports.py is used by hand in the future, I think it should be executed also with umask or sudo -u discovery-stats.
@Gehel
As main autor of reportupdater, I totally agree with you on that reportupdater should not create files in source directories. Created this task to solve that: T173604. Thanks!
Mentioned in SAL (#wikimedia-operations) [2017-08-23T18:40:28Z] <gehel> resetting permissions on stat1005:/srv/published-datasets/discovery - T173333
Mentioned in SAL (#wikimedia-operations) [2017-08-23T18:42:59Z] <gehel> manually running report updater for discovery golden data on stat1005 - T173333
Change 373506 had a related patch set uploaded (by Gehel; owner: Gehel):
[wikimedia/discovery/golden@master] Set umask to ensure files are created with correct permissions.
Mentioned in SAL (#wikimedia-operations) [2017-08-24T12:45:39Z] <gehel> killing discovery report updater on stats1005 (stuck since Aug 15) - T173333
Change 373506 merged by Bearloga:
[wikimedia/discovery/golden@master] Set umask to ensure files are created with correct permissions.
Change 371955 abandoned by Bearloga:
Give group write permission to output files
Reason:
Abandoning in favor of the umask solution. Thanks everyone!