Page MenuHomePhabricator

[Airflow] SparkSqlOperator fails when executing via Skein with master=local
Closed, InvalidPublic

Description

When working on T352948, I realized that SparkSqlOperator doesn't work in production if launcher=skein (default) and master=local.
I didn't dig deep into the error, but the error message just seemed to indicate that the query was wrong. Which is surprising, since the query was fine.
Executing the DAG from the development instance worked, since it doesn't use Skein.
In production, just by removing the master=local flag, the operator started succeeding.

I think we should look into this, in case we want to execute with launcher=skein and master=local in the future.

Event Timeline

We currently have use-cases doing this exactly that work. there must have been another issue than the pone described here. I think this ticket is invalid.

lbowmaker subscribed.

Based on @JAllemandou comment I will close. @mforns let us know if you want us to investigate further.