Diffusion wikimedia/discovery/analytics d1f43675ad79

weekly import of image suggestions
d1f43675ad79
Actions

Description

weekly import of image suggestions

Image suggestions are now available on a weekly basis from hive tables.
Unfortunately the weekly schedules don't line up, one of the two has to
wait for the other. For now this is setup to read the image suggestions
that are available when the weekly execution starts.

Bug: T307983
Change-Id: I2ab3ee488bceba3438f7225ee429fcc563d1f9ea

Details

Provenance

EBernhardson

Authored on May 23 2022, 5:43 PM

Parents

rWDANb5b63c3fe32a: wdqs reconciliation: fix entity to namespace map

Branches

Unknown

Tags

Unknown

ChangeId

I2ab3ee488bceba3438f7225ee429fcc563d1f9ea

Event Timeline

EBernhardson committed rWDANd1f43675ad79: weekly import of image suggestions (authored by EBernhardson).May 23 2022, 5:58 PM

Changes (7)

Path

Size

airflow/

dags/

transfer_to_es.py

tests/

conftest.py

fixtures/

skein_operator_spec/

	image_suggestions_manual-upload_to_swift.expected
	imagerec_manual-upload_to_swift.expected

imagerec_manual-upload_to_swift.expected

spark_submit_operator/

	image_suggestions_manual_convert_to_esbulk.expected
	imagerec_manual_convert_to_esbulk.expected

imagerec_manual_convert_to_esbulk.expected

spark/

convert_to_esbulk.py

rWDANd1f43675ad79

View Options

airflow/dags/transfer_to_es.py

View Options

airflow/tests/conftest.py

View Options

airflow/tests/fixtures/skein_operator_spec/image_suggestions_manual-upload_to_swift.expected

View Options

airflow/tests/fixtures/skein_operator_spec/imagerec_manual-upload_to_swift.expected

View Options

airflow/tests/fixtures/spark_submit_operator/image_suggestions_manual_convert_to_esbulk.expected

View Options

airflow/tests/fixtures/spark_submit_operator/imagerec_manual_convert_to_esbulk.expected

View Options

spark/convert_to_esbulk.py

weekly import of image suggestionsd1f43675ad79Actions

Description

Details

Event Timeline

Changes (7)

rWDANd1f43675ad79

airflow/dags/transfer_to_es.py

airflow/tests/conftest.py

airflow/tests/fixtures/skein_operator_spec/image_suggestions_manual-upload_to_swift.expected

airflow/tests/fixtures/skein_operator_spec/imagerec_manual-upload_to_swift.expected

airflow/tests/fixtures/spark_submit_operator/image_suggestions_manual_convert_to_esbulk.expected

airflow/tests/fixtures/spark_submit_operator/imagerec_manual_convert_to_esbulk.expected

spark/convert_to_esbulk.py

weekly import of image suggestions
d1f43675ad79
Actions