@xcollazo , what about setting paths with VariableProperties, pretty much as we do with the conda artifact?
Something like helper = var_props.get('helper', '/path/to/hdfs')

In T324125#8530275, @mfossati wrote:

@xcollazo , what about setting paths with VariableProperties, pretty much as we do with the conda artifact?
Something like helper = var_props.get('helper', '/path/to/hdfs')

Yes, that makes sense.

I had two concerns with the current way of using static files:

1. what happens if we migrate the DAG elsewhere?
2. what happens if someone else starts using the platform_eng Airflow instance and inadvertedly messes up the files

@mfossati suggestion takes care of (1), as we could just override temporarily while we change the default to the new place.

For (2): We can also come up with some namespacing strategy like:

/user/analytics-platform-eng/structured-data/image_suggestions/ for image_suggestions static files

and

/user/analytics-platform-eng/structured-data/section_topics/ for section_topics static files

etc.

I think the above would make it clear that these static files should not be touched by other folks.

In T324125#8540703, @xcollazo wrote:

1. what happens if we migrate the DAG elsewhere?
2. what happens if someone else starts using the platform_eng Airflow instance and inadvertedly messes up the files

@mfossati suggestion takes care of (1), as we could just override temporarily while we change the default to the new place.

For (2): We can also come up with some namespacing strategy like:

/user/analytics-platform-eng/structured-data/image_suggestions/ for image_suggestions static files

and

/user/analytics-platform-eng/structured-data/section_topics/ for section_topics static files

etc.

I think the above would make it clear that these static files should not be touched by other folks.

That sounds good to me!