Page MenuHomePhabricator

Failed to find any Kerberos tgt
Closed, ResolvedPublic

Description

I'm trying to see my application logs by SSH'ing into airflow1002 and running:

sudo -u analytics-research yarn logs -appOwner analytics-research -applicationId application_1663082229270_26382

But I'm getting this error:

Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

A similar command used to work fine before. So I tried destroying and re-initializing the kerberos key with sudo -u analytics-research kdestroy and sudo -u analytics-research kinit. The destroy part worked, but init isn't working. Looks like I messed something up while trying to fix the above issue. Any help?

Event Timeline

Yep, as @otto indicated, could you check by running the following please?

sudo -u analytics-research kerberos-run-command analytics-researh yarn logs -appOwner analytics-research -applicationId application_1663082229270_26382

Yep, as @otto indicated, could you check by running the following please?

sudo -u analytics-research kerberos-run-command analytics-researh yarn logs -appOwner analytics-research -applicationId application_1663082229270_26382

Here's what I'm getting:

bmansurov@an-airflow1002:~$ sudo -u analytics-research kerberos-run-command analytics-researh yarn logs -appOwner analytics-research -applicationId application_1663082229270_26382
The user keytab that you are trying to use (/etc/security/keytabs/analytics-researh/analytics-researh.keytab) doesn't exist or it isn't readable from your user, aborting...

I'll check out the link Andrew shared.

This seems to have fixed the issue:

bmansurov@an-airflow1002:~/research$ sudo -u analytics-research kerberos-run-command analytics-research klist
Ticket cache: FILE:/tmp/krb5cc_912
Default principal: analytics-research/an-airflow1002.eqiad.wmnet@WIKIMEDIA

Valid starting       Expires              Service principal
09/29/2022 03:42:42  10/01/2022 03:42:42  krbtgt/WIKIMEDIA@WIKIMEDIA
	renew until 10/06/2022 03:42:42
bmansurov@an-airflow1002:~/research$ sudo -u analytics-research yarn logs -appOwner analytics-research -applicationId application_1663082229270_26382
# The above command works.

Thank you, both!