After review of our code and documentation, industry best practices, and internal discussion, I propose to adopt the following principles as part of our design and practices for generic dags.

Design principles

We should make sure that tasks are reproducible and follow functional programming paradigms:

• Tasks should be idempotent.
• Tasks should be deterministic (see ).
• Tasks should not generate side effects.

We should provide documentation and guidelines with examples of how to achieve that.
@lbowmaker @Clarakosi maybe we could make a documentation phab task out of this?

Implementation details

From an implementation perspective, dag should follow this principles:

• DAGs should be a thin layer and not contain any business logic, queries, computation etc.
• Specify config details consistently, by moving parameters to a config file.
• Group tasks in the Airflow UI, to make it explicit to users the status of data processing / enrichment / export steps of a data pipeline.
• Avoid local computations (our airflow instance shares scheduler and task executor processes). Currently we target the YARN resource manager of elastic compute. Data pipeline are implemented atop Apache Spark. Spark jobs must be submitted from the airflow instance, and run in cluster mode.

image-matching

In order to embrace the above principles, we should refactor image-matching to:

• Cleanup DAGs init parameters and move them to a config file
• Remove local path references (/srv/...)
• Make sure algo runs are deterministic (done in https://phabricator.wikimedia.org/T280585)
• Remove intermediate algo result caching
• Do not store task output locally (done in https://phabricator.wikimedia.org/T292740)
• Submit spark jobs in cluster mode (done in https://phabricator.wikimedia.org/T292740) to remove local computation
• Submit SQL jobs using the spark-sql shell instead of hive

These changes depend on https://phabricator.wikimedia.org/T292740 and https://phabricator.wikimedia.org/T280585. Once those tasks are merged, refactoring the rest will be trivial.

Further improvements

This could be split into a separate task, but we should

• Use DAG factories to template common topologies. I have a conceptual demo at https://gitlab.wikimedia.org/gmodena/platform-airflow-dags/-/blob/T292748-dag-factory/dags/ima_v2.py

WIP: image-matching refactoring is in progress at https://gitlab.wikimedia.org/gmodena/platform-airflow-dags/-/merge_requests/19

@gmodena / @lbowmaker: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!