Research 's code repository for image suggestions based on section alignment contains an Airflow DAG: the section-level image suggestions Airflow job should take it as a dependency, i.e., wait for its completion, since its output will serve as input for the final data pipeline.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T311814 [EPIC] Section-level image suggestions data pipeline | |||
| Resolved | xcollazo | T328641 Productionize the Airflow DAG of section alignment-based suggestions |
FYI @xcollazo: I ran the Research's repo in a stat machine with Spark 3 as follows:
conda-analytics-clone dev source conda-analytics-activate dev pip install mwparserfromhell
These changes made it work:
from wmfdata.spark import create_session
spark = create_session(
type='yarn-large',
ship_python_env=True,
extra_settings={ 'spark.sql.shuffle.partitions': 1024 },
)from dataclasses import dataclass @dataclass class SectionImages:
xcollazo updated https://gitlab.wikimedia.org/repos/structured-data/section-image-recs/-/merge_requests/1
Bump to Spark3. Use conda for dep management. Use poetry for build and test.
xcollazo merged https://gitlab.wikimedia.org/repos/structured-data/section-image-recs/-/merge_requests/1
Bump to Spark3. Use conda for dep management. Use poetry for build and test.
xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/228
Draft: T328641 add section image recs dag
Addressed review suggestions to https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/228.
Asked for final review.
xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/228
T328641 add section image recs dag