Page MenuHomePhabricator

Set hive.warehouse.subdir.inherit.perms to false
Closed, ResolvedPublic

Description

In T280175: Fix default ownership and permissions for Hive managed databases in /user/hive/warehouse we changed the permissions on the /user/hive/warehouse directory, expecting to take advantage of our HDFS fs.permissions.umask-mode setting, so that the following will hold:

  • new databases in /user/hive/warehouse are writeable by owner, readable by analytics-privatedata-users group, and unreadable by other.
  • all sub directories of /user/hive/warehouse inherit the same permissions.

However, in order to allow anyone to create new Hive databases (like we always have), we need to make /user/hive/warehouse writeable by the analytics-privatedata-users group. I had expected this to work if we set the permissions manually, however, our version of Hive has a hive.warehouse.subdir.inherit.perms setting that overrides the HDFS umask, and will cause any new sub directories to inherit the parent's.

We should set hive.warehouse.subdir.inherit.perms to false to avoid this behavior.

See

Event Timeline

Ottomata triaged this task as Medium priority.
Ottomata added a project: Analytics-Kanban.
Ottomata moved this task from Backlog to Q2 2021/2022 on the Analytics-Clusters board.
Ottomata updated the task description. (Show Details)

Change 734368 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Hive - set hive.warehouse.subdir.inherit.perms = false

https://gerrit.wikimedia.org/r/734368

Change 734368 merged by Ottomata:

[operations/puppet@production] Hive - set hive.warehouse.subdir.inherit.perms = false

https://gerrit.wikimedia.org/r/734368

Mentioned in SAL (#wikimedia-analytics) [2022-02-11T15:06:49Z] <ottomata> set hive.warehouse.subdir.inherit.perms = false - T291664

Change 768787 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Revert "Hive - set hive.warehouse.subdir.inherit.perms = false"

https://gerrit.wikimedia.org/r/768787

Change 768787 merged by Ottomata:

[operations/puppet@production] Revert "Hive - set hive.warehouse.subdir.inherit.perms = false"

https://gerrit.wikimedia.org/r/768787

Mentioned in SAL (#wikimedia-analytics) [2022-03-07T18:34:33Z] <ottomata> restarting hive-server2 on an-coord1001 to revert hive.warehouse.subdir.inherit.perms change - T291664

Mentioned in SAL (#wikimedia-analytics) [2022-03-07T18:37:04Z] <ottomata> sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chgrp -R analytics-privatedata-users /wmf/data/wmf/webrequest/webrequest_source=text/year=2022/month=3/day=7 - after reverting - T291664

Mentioned in SAL (#wikimedia-analytics) [2022-03-07T19:13:30Z] <ottomata> sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chgrp -R analytics-privatedata-users /wmf/data/wmf/virtualpageview/hourly/year=2022/month=3/day=7 - revert of T291664

Mentioned in SAL (#wikimedia-analytics) [2022-03-07T19:14:45Z] <ottomata> sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chgrp -R analytics-privatedata-users /wmf/data/wmf/*/hourly/year=2022/month=3/day=7 to make sure perms are fixed after revert of T291664

I had to revert this change, as it somehow broke group ownership for some of the files our oozie loading jobs create.

Not sure how to proceed on this then. I'm inclined to just leave things as they are. This would be:

  • new Hive databases and tables will be 0770 owner:analytics-privatedata-users. We were aiming for 0750 owner:analytics-privatedata-users. I suppose 0770 is okay, it just means that users in analytics-privatedata-users can mess with each others files. Likely in practice that won't matter too much.

Mentioned in SAL (#wikimedia-analytics) [2022-03-09T16:16:33Z] <ottomata> fix group ownership of wmf_product.db/poageviews_corrected/year=222/month=2 after reverting T291664 - sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chgrp -R analytics-privatedata-users /user/hive/warehouse/wmf_product.db/pageviews_corrected/year=2022/month=2

Mentioned in SAL (#wikimedia-analytics) [2022-03-09T18:32:30Z] <ottomata> fix group ownership of wmf_product.db/global_markets_pageviews/year=2022/month=2 after reverting T291664 - sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chgrp -R analytics-privatedata-users /user/hive/warehouse/wmf_product.db/global_markets_pageviews/year=2022/month=2

Mentioned in SAL (#wikimedia-analytics) [2022-03-09T18:33:40Z] <ottomata> fix group ownership of wmf_product.db//new_editors/cohort=2021-12 after reverting T291664 - sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chgrp -R analytics-privatedata-users /user/hive/warehouse/wmf_product.db/new_editors/cohort=2021-12

Mentioned in SAL (#wikimedia-analytics) [2022-03-09T21:05:54Z] <ottomata> fix group ownership of cchen.db/new_editors/cohort=2021-12 after reverting T291664 - sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chgrp -R analytics-privatedata-users /user/hive/warehouse/cchen.db/new_editors/cohort=2021-12