Page MenuHomePhabricator

Append environment for BashSensor
Closed, InvalidPublic

Description

The Airflow BashSensor module is lacking (Airflow#31221, don't look) an append_env parameter so any environment passed to it will overwrite the executor's default environment. In our case, we need the KRB5CCNAME and other Kerberos variables which are set up by our Airflow configuration. I've attempted to wire the DAG parse environment through to the sensor, which works in the development environment, but not in production. What I found is that the credentials cache is set to /tmp/airflow_krb5_ccache/krb5cc in the DAG parse context, but should be /tmp/airflow_krb5_ccache for the executor.

This task is about merging passed task environment with the default executor environment.

Event Timeline

Wrote this too quickly—after looking into it more I see that the BashSensor actually uses the same krb5 credentials path cache so my shortcut *does* work.

It's the SimpleSkeinOperator which has a problem and that can be solved by copying less environment into the append_env.

It's the SimpleSkeinOperator which has a problem and that can be solved by copying less environment into the append_env.

Do you depend on this?