mnz opened https://gitlab.wikimedia.org/repos/research/revertrisk-datasets/-/merge_requests/1

Draft: Add spark job for generating revertrisk multilingual datasets

mnz merged https://gitlab.wikimedia.org/repos/research/revertrisk-datasets/-/merge_requests/1

Add spark job for generating revertrisk multilingual datasets

The first dataset generation pipeline (https://gitlab.wikimedia.org/repos/research/revertrisk-datasets) is complete, including the corresponding airflow dag (https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/research/dags/revert_risk_multilingual_dataset_dag.py). For ML training datasets, this is a good starting point to collect more experience to answer open questions in the description.

Research engineering is starting work on two related projects:

• T343065 is a pipeline that creates a dataset is used to populate a dashboard
• A pipeline to create embedding datasets for use in projects based on vector similarity search

This has lead to additional questions around code organization and re-use of code, beyond the use case of machine learning pipelines, the answers which will be useful for this task as well. Pushing back the due date as we will revisit this task after progress has been made on the related work mentioned above.

@MunizaA / @fkaelin: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!