Improvements of artifacts cache
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Antoine_Quhen
	Apr 28 2022, 3:09 PM

Description

Improvements:

create an independent cache store by airflow instance
warn of unused artifacts when running workflow_utils/artifact/cli/warm By listing the cache and diffing with the yaml file
add an evict script (to be used in airflow-dags) to clean the cache from unspecified artifacts (the ones removed from artifacts.yaml)
Maybe move this artifact caching library into its own repo
Cached artifacts from Gitlab package 'download' links that are 'archives' (e.g. .tgz files) don't work with SparkSubmitOperator archives param. This param expects archive files to end in an extension like .tgz in order to automate unpacking the archive on the workers. This should be fixed. See: this MR: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/47#note_6838. Done in MR25

	Subject	Repo	Branch	Lines +/-
	Release 2.1.4-py3.7-5	operations/debs/airflow	debian	+12 -5

Title	Reference	Author	Source Branch	Dest Branch
Rename artifacts, these are now cached by name, so use apporpriate file ext	repos/data-engineering/airflow-dags!58	otto	artifact-naming	main
Fix bug in FsArtifactSource where empty base_uri would cause wrong fsspec fs to be used	repos/data-engineering/workflow_utils!27	otto	artifact-source-fs	main
Make ArtifactLocator methods work with Artifacts, rather than artifact ids, use artifact.name as default cache_key	repos/data-engineering/workflow_utils!25	otto	artifact	main

Thanks Antoine! I just added another needed fix too.

Change 793504 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/debs/airflow@debian] Release 2.1.4-py3.7-5

Mentioned in SAL (#wikimedia-analytics) [2022-05-19T16:59:53Z] <ottomata> deploying airflow-dags analytics with new artifact names, first clearing artifacts cache dir - T307115

Change 793504 abandoned by Ottomata:

[operations/debs/airflow@debian] Release 2.1.4-py3.7-5

Reason: