The image suggestions data pipeline should follow the Wikidata snapshot release schedule, i.e., weekly on Mondays: this would let the pipeline leverage the freshest data.

However, a manual execution of the Airflow job resulted in empty commonswiki_file.py output for the 2022-04-11 Wikidata snapshot.
The main suspects are joins between weekly Wikidata snapshots and monthly Wikis ones. For instance, if we are at the beginning of April, Wikidata would have a snapshot on 2022-04-04, but maybe other Wikis are still on March?
This requires further investigation to answer the following questions:

• is the Airlfow Hive sensor that waits for the latest Wikidata snapshot still useful?
• Should we agree on a different schedule that ensures workable data?

Update: solution

We agreed with @JAllemandou that the best solution is to let the Airflow Hive sensor wait for the latest available snapshot (read partition) of every table we query in the pipeline. This would ensure the freshest data.
Usually wmf_raw table need 2-3 days of processing before they become available, but there may be an additional delay for unexpected reasons.
As a result, we set a timeout of 6 days before the sensor stops, so that the next scheduled pipeline run can be still be triggered weekly.
See https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines/-/merge_requests/55