Page MenuHomePhabricator

Investigate how Airflow dynamic task mapping works with traditional operators like SparkSqlOperator
Open, Needs TriagePublic

Description

When working on T375153 (MR (846)), I developed a DAG to dynamically fetch the list of wikis where Automoderator is deployed, and get the data from the corresponding MariaDB-replicas, for which I had to use dynamic task mapping. I also needed generate unique output paths for each database. Initially, I used f-strings, like tmp_file_path=f"{props.tmp_directory}/{wiki_db}" - this was causing an issue during testing, where the whole MapppedArgument object was being passed, instead of resolved wiki_db string value. The workaround that worked was, using XCom variables and jinja templates.

During the MR review, @xcollazo suggested to further investigate this behaviour.

Event Timeline

KCVelaga_WMF moved this task from Triage to Tracking on the Product-Analytics board.

@xcollazo @amastilovic created an initial ticket from our conversation on GitLab, please adjust the title, description and what needs to be done as necessary.