Page MenuHomePhabricator
Create Task
Maniphest T398907

Spark Job in airflow-devenv cannot access Hive Metastore because of Kerberos Authentication Failure
Closed, ResolvedPublic
Actions

Description

In T396495#10941826, we ran a PoC ML training Airflow pipeline end-to-end using dummy data.

As a next step, I have been working on adapting this pattern to run an ML training Airflow pipeline with real-world data. Specifically, I worked on a simple add_a_link Airflow pipeline with the following components:

However, when running this pipeline in the airflow-devenv, the DAG fails due to a Kerberos authentication error when attempting to connect to the Hive metastore. This prevents the Spark job from accessing the required Hive tables.

For reference, here are the detailed YARN logs from the failed runs: https://phabricator.wikimedia.org/P78736

Details

Other Assignee
kevinbazira
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
airflow-ml: enable the hadoop shellBrouberoloperations/deployment-chartsmaster+0 -3
airflow-ml: update the principal primary to analyticsBrouberoloperations/deployment-chartsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

kevinbazira created this task.Jul 8 2025, 4:48 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJul 8 2025, 4:48 AM
kevinbazira added a comment.Jul 8 2025, 5:29 AM

To isolate the issue, I ran a manual test to confirm whether direct Hive access works from a Kerberos-authenticated Spark client outside of the airflow-devenv. Below are the steps I took, which successfully queried the wmf.wikidata_entity table:

$ ssh stat1008.eqiad.wmnet

$ kinit

$ conda-analytics-clone test-hive-access-env
$ source conda-analytics-activate test-hive-access-env

$ echo 'from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("test_hive_access").getOrCreate()
df = spark.table("wmf.wikidata_entity")
df.show(5)
spark.stop()' > test_hive_access.py

$ spark-submit --master yarn test_hive_access.py

...
+---------+----+--------+--------------------+--------------------+-------+--------------------+---------+----------+----------+
|       id| typ|datatype|              labels|        descriptions|aliases|              claims|siteLinks| lastrevid|  snapshot|
+---------+----+--------+--------------------+--------------------+-------+--------------------+---------+----------+----------+
|Q39457913|item|    null|{en -> "Different...|{ka -> "სამეცნიერ...|   null|[{Q39457913$75C27...|     null|2302400758|2025-05-12|
|Q82745689|item|    null|{en -> "MR urogra...|{hy -> "գիտական հ...|   null|[{Q82745689$3A5B6...|     null|2222889061|2025-05-12|
|Q38759012|item|    null|{en -> "Methods f...|{hy -> "2011 թվակ...|   null|[{Q38759012$FD0C4...|     null|2282918747|2025-05-12|
|Q80324703|item|    null|{bn -> "ওগল স্ফীত...|{ar -> "نجم متغير...|   null|[{Q80324703$54EF3...|     null|2327488889|2025-05-12|
|Q38954651|item|    null|{en -> "Mental re...|{hy -> "գիտական հ...|   null|[{Q38954651$5E0B7...|     null|2219836121|2025-05-12|
+---------+----+--------+--------------------+--------------------+-------+--------------------+---------+----------+----------+
only showing top 5 rows

...
isarantopoulos moved this task from Unsorted to Blocked on the Machine-Learning-Team board.Jul 8 2025, 1:52 PM
gerritbot added a comment.Jul 9 2025, 7:28 AM

Change #1167508 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-ml: update the principal primary to analytics

https://gerrit.wikimedia.org/r/1167508

gerritbot added a project: Patch-For-Review.Jul 9 2025, 7:28 AM

Change #1167509 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-ml: enable the hadoop shell

https://gerrit.wikimedia.org/r/1167509

gerritbot added a comment.Jul 9 2025, 7:55 AM

Change #1167508 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-ml: update the principal primary to analytics

https://gerrit.wikimedia.org/r/1167508

gerritbot added a comment.Jul 9 2025, 7:55 AM

Change #1167509 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-ml: enable the hadoop shell

https://gerrit.wikimedia.org/r/1167509

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2025, 8:30 AM
kevinbazira added a comment.Jul 10 2025, 3:02 PM

Thanks to @brouberol, who has been super helpful when resolving WMF Airflow issues. He also provided clarity on where our pipelines will run:

  • devenv: runs with a users own kinit credentials
  • production: runs with its own dedicated kerberos keytab

The issue detailed in this ticket's description where spark jobs fail to access the Hive Metastore is caused by a kerberos authentication failure in the airflow-devenv.

As Balthazar is debugging this issue, he configured the airflow-ml production environment to run using my development branch and suggested testing it there, since production uses a dedicated kerberos keytab. The simple add_a_link ETL pipeline now runs successfully in the the airflow-ml instance and can access real-world data from the WMF datalake.

brouberol added a comment.Jul 10 2025, 7:36 PM

I have merged a change into airflow-dags (see https://phabricator.wikimedia.org/T394297#10993349) that should resolve the issue of the skein/spark job not being able to connect to hive when run from a devenv. Would you be able to report back whether that now works as expected?

kevinbazira added a comment.Jul 11 2025, 1:50 AM
In T398907#10993364, @brouberol wrote:

I have merged a change into airflow-dags (see https://phabricator.wikimedia.org/T394297#10993349) that should resolve the issue of the skein/spark job not being able to connect to hive when run from a devenv. Would you be able to report back whether that now works as expected?

Really appreciate all the fixes. I tried running the pipeline again from this branch in a clean devenv setup, but unfortunately the same error persists:

25/07/11 01:09:25 INFO metastore: Trying to connect to metastore with URI thrift://analytics-hive.eqiad.wmnet:9083
25/07/11 01:09:25 ERROR TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

I have uploaded the full logs here for reference: https://phabricator.wikimedia.org/P78736#316863

kevinbazira added a comment.Jul 17 2025, 2:36 AM

After the fix in: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1561

I ran the pipeline again using this branch in a newly created devenv, and it was able to access the hive metastore:

25/07/17 02:12:38 INFO metastore: Trying to connect to metastore with URI thrift://analytics-hive.eqiad.wmnet:9083
25/07/17 02:12:38 INFO metastore: Opened a connection to metastore, current connections: 1
25/07/17 02:12:38 INFO metastore: Connected to metastore.
25/07/17 02:12:38 INFO BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://analytics-hadoop/wmf/data/wmf_content/mediawiki_content_current_v1/metadata/00124-1c3cf735-30bc-49e3-8ace-a1ce66ec2cb3.metadata.json

Thanks a lot @brouberol, for fixing this.

kevinbazira closed this task as Resolved.Aug 12 2025, 5:38 AM
kevinbazira assigned this task to brouberol.
kevinbazira updated Other Assignee, added: kevinbazira.
kevinbazira moved this task from Blocked to 2025-2026 Q1 Done on the Machine-Learning-Team board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits