Page MenuHomePhabricator

profile::hive::site_hdfs should work with kerberos-run-command
Closed, ResolvedPublic3 Estimated Story Points

Description

Two problems:

  • /user/hive is not writable by the analytics user
  • kerberos::exec wrapper does not work with multiple shell commands, e.g. cmd1 && cmd2 && cmd3

We need to run an hdfs dfs -put command from puppet somewhere whenever hive-site.xml changes. This can't be done from an-coord1001, because hive-site.xml there has password in it.

Event Timeline

Ottomata created this task.Dec 16 2019, 5:33 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 16 2019, 5:33 PM
fdans assigned this task to elukey.Dec 23 2019, 4:56 PM
fdans triaged this task as High priority.
fdans moved this task from Incoming to Operational Excellence on the Analytics board.
elukey moved this task from Backlog to Kerberos on the User-Elukey board.Jan 3 2020, 10:53 AM

Change 573237 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hive::site_hdfs: fix exec using bash -c to execute commands

https://gerrit.wikimedia.org/r/573237

Change 573237 merged by Elukey:
[operations/puppet@production] profile::hive::site_hdfs: fix exec using bash -c to execute commands

https://gerrit.wikimedia.org/r/573237

The only issue left should be to enable the exec on a host with hive-site.xml and the hdfs keytab. I propose to elect as candidate the new an-jobrunner1001 (or whatever we are gonna call it :).

Change 574044 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::launcher: set hive-site.xml in HDFS

https://gerrit.wikimedia.org/r/574044

Change 574044 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::launcher: set hive-site.xml in HDFS

https://gerrit.wikimedia.org/r/574044

elukey set the point value for this task to 3.Mar 6 2020, 7:40 AM
elukey moved this task from Next Up to Done on the Analytics-Kanban board.
Nuria closed this task as Resolved.Apr 9 2020, 5:11 PM