mfossati opened https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/38

Compute the search index delta against the discovery.cirrus_index_without_content Hive table

Approved the MR

Code looks good to me

Looking into the discrepancies between the old diff and the new diff ...

In total there are 1020 rows in the old diff that are not in the new:

• 327 rows refer to deleted pages - which we can ignore because these will be dropped by the cirrus import job
• select 10 rows at random from the remaining 693 not-deleted pages, and for all of them the data is already in cirrus search

In total there are 195903 rows in the new diff that are not in the old:

• 13448 rows refer to deleted pages - which we can ignore because these will be dropped by the cirrus import job
• Excluding the rows referring to deleted pages
  • 158937 are __DELETE_GROUPING__ instructions (deleting the relevant weighted_tag from the search index), and none of the pages appear in image_suggestions_search_index_full so it makes sense to delete them (note that 146945 of these are in the commons index)
  • 23478 have values for weighted_tags, and there's a row in image_suggestions_search_index_full corresponding to every one, so it makes sense to write them

So overall it looks very likely that the discrepencies are errors that crept in due to imports that were incomplete or re-done or aborted, and this new code corrects them (which is exactly what we were hoping for)

Based on the above there's a chance that we'll get alerts the first time this code is run because there's quite a few deletions (of things that probably should been deleted before) , but they're mostly on commons so perhaps not

mfossati merged https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/38

Compute the search index delta against the discovery.cirrus_index_without_content Hive table

Note that a DAG update would be nice, as we don't need the previous snapshot anymore.

Moving back to doing: DAG update & Airflow test run.

Airflow test run's output:

snapshot = '2024-01-08'
prod = spark.read.table('analytics_platform_eng.image_suggestions_search_index_delta').where(f'snapshot="{snapshot}"')
dev = spark.read.table('is_new_delta.image_suggestions_search_index_delta').where(f'snapshot="{snapshot}"')
prod.count(), dev.count()

(130903, 322933)

Row counts seem in line with https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/38#caveat

DAG merge request ready at https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/585.
Moving to code review again.

mfossati merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/585

Bump image-suggestions to 0.17.0-v0.17.0.

Deployed, moving to verify on production.

This week verification was impossible due to a missing upstream dependency (wmf.wikidata_item_page_link/snapshot=2024-01-15).
If the DAG times out, it will be a good opportunity to test the intended robustness of this ticket.

The DAG timed out as expected. I think we can just wait for the next to be picked up.

See also T356030: Search dag image_suggestions_weekly failed waiting for analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15.