Change Details

When working on T375153 ([[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/846 | MR (846) ]]), I developed a DAG to dynamically fetch the list of wikis where Automoderator is deployed, and get the data from the corresponding MariaDB-replicas, for which I had to use dynamic task mapping. I also needed generate unique output paths for each database. Initially, I used f-strings, like `tmp_file_path=f"{props.tmp_directory}/{wiki_db}"` - this was causing an issue during testing, where the whole MapppedArgument object was being passed, instead of resolved `wiki_db` string value. The workaround that worked was, using XCom variables and jinja templates. During the MR review, @xcollazo suggested to further investigate behaviour.

When working on T375153 ([[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/846 | MR (846) ]]), I developed a DAG to dynamically fetch the list of wikis where Automoderator is deployed, and get the data from the corresponding MariaDB-replicas, for which I had to use dynamic task mapping. I also needed generate unique output paths for each database. Initially, I used f-strings, like `tmp_file_path=f"{props.tmp_directory}/{wiki_db}"` - this was causing an issue during testing, where the whole MapppedArgument object was being passed, instead of resolved `wiki_db` string value. The workaround that worked was, using XCom variables and jinja templates. During the MR review, @xcollazo suggested to further investigate this behaviour.

When working on T375153 ([[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/846 | MR (846) ]]), I developed a DAG to dynamically fetch the list of wikis where Automoderator is deployed, and get the data from the corresponding MariaDB-replicas, for which I had to use dynamic task mapping. I also needed generate unique output paths for each database. Initially, I used f-strings, like `tmp_file_path=f"{props.tmp_directory}/{wiki_db}"` - this was causing an issue during testing, where the whole MapppedArgument object was being passed, instead of resolved `wiki_db` string value. The workaround that worked was, using XCom variables and jinja templates. During the MR review, @xcollazo suggested to further investigate this behaviour.