Maniphest T337410

The data check task in the image suggestions DAG is failing
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Cparle
	May 24 2023, 3:43 PM

Project Tags

Referenced Files

None

Subscribers

Description

The problem is the script doesn't use spark, but is called with SparkSubmitOperator. It needs to be called with SimpleSkeinOperator instead

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Use SimpleSkeinOperator instead of SparkSubmitOperator for data check	repos/data-engineering/airflow-dags!406	cparle	T337410	main

Customize query in GitLab

Event Timeline

Cparle created this task.May 24 2023, 3:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 24 2023, 3:43 PM

Cparle moved this task from Incoming to Doing on the Structured-Data-Backlog (Current Work) board.May 24 2023, 3:43 PM

CBogen added a project: Image-Suggestions.May 25 2023, 1:36 PM

cparle opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/406

Use SimpleSkeinOperator instead of SparkSubmitOperator for data check

CodeReviewBot added a project: Patch-For-Review.May 31 2023, 8:31 AM

mfossati merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/406

Use SimpleSkeinOperator instead of SparkSubmitOperator for data check

Maintenance_bot removed a project: Patch-For-Review.May 31 2023, 10:10 AM

Cparle moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.May 31 2023, 1:35 PM

mfossati closed this task as Resolved.May 31 2023, 4:52 PM