Search dag image_suggestions_weekly failed waiting for analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcausse
	Jan 29 2024, 8:28 AM

Project Tags

Referenced Files

None

Subscribers

Description

This dataset (analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15) is not available and caused the search dag image_suggestions_weekly to fail with a timeout and will have to be restarted once the source data is available.

Root cause was identified and first raised by @mfossati via slack:

wmf.wikidata_item_page_link/snapshot=2024-01-15 is not available

According to @Antoine_Quhen this partition is missing because some sensors on event.mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=21 timed out (related to problem with canary events: T337055#9470093)

This task is to coordinate the efforts regarding backfilling the datasets:

wmf.wikidata_item_page_link/snapshot=2024-01-15
analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15
image_suggestions_weekly@2024-01-22T00:00:00+00:00 is retried and suggestions are shipped to elasticsearch

~~NOTE: blocked by T356400: User aqsloader hasn't MODIFY permissions on image_suggestions.* Cassandra tables anymore.~~

Details

Other Assignee: Antoine_Quhen

Related Objects

Mentioned In: T338013: [L] Create search index deltas by comparing to `discovery.cirrus_index_without_content` in hive
Mentioned Here: T356400: User aqsloader hasn't MODIFY permissions on image_suggestions.* Cassandra tables anymore
T338013: [L] Create search index deltas by comparing to `discovery.cirrus_index_without_content` in hive
T337055: Send a critical alert to data-engineering if produce_canary_events isn't running correctly

Event Timeline

dcausse created this task.Jan 29 2024, 8:28 AM

Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJan 29 2024, 8:28 AM

dcausse updated the task description. (Show Details)Jan 29 2024, 8:29 AM

dcausse updated the task description. (Show Details)Jan 29 2024, 8:32 AM

dcausse updated the task description. (Show Details)Jan 29 2024, 1:33 PM

dcausse added a subscriber: Antoine_Quhen.

gmodena subscribed.Jan 29 2024, 2:06 PM

Antoine_Quhen edited projects, added: Data-Engineering (Sprint 7); removed: Data-Engineering.Jan 30 2024, 9:49 AM

mfossati mentioned this in T338013: [L] Create search index deltas by comparing to `discovery.cirrus_index_without_content` in hive.Jan 30 2024, 10:46 AM

I've added the missing partitions in the source of wikidata_item_page_link and its missing snapshot is now generated.

# check for missing folders on 2 days
hdfs dfs -ls /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/
# 8 partitions missing
hdfs dfs -ls /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=15/
# 1 partition missing


# Check on raw data
hdfs dfs -ls /wmf/data/raw/event/eqiad.mediawiki.page-move/year=2024/month=01/day=17/
# same missing imports. Probably a canary event problem

# Trying to add the raw folders manually and re-import, without success at it needs at least 1 event to guess the schema.
# refine_event implements a lock for only 1 process to run at a time, so we can't run it in parallel.
# This costs me time in my debugging...

# add missing folders in event dir
sudo -u analytics kerberos-run-command analytics \
  hdfs dfs -mkdir -p \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=14 \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=15 \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=16 \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=17 \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=18 \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=19 \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=20 \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/hour=21 \
  /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=15/hour=11

# check
hdfs dfs -ls /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=17/
hdfs dfs -ls /wmf/data/event/mediawiki_page_move/datacenter=eqiad/year=2024/month=1/day=15/

# repair table to add partitions matching folders & check before/after
spark3-sql -e "show partitions event.mediawiki_page_move;" | grep "datacenter=eqiad/year=2024/month=1/day=17"
spark3-sql -e "show partitions event.mediawiki_page_move;" | grep "datacenter=eqiad/year=2024/month=1/day=15"
spark3-sql -e "msck repair table event.mediawiki_page_move;"
spark3-sql -e "show partitions event.mediawiki_page_move;" | grep "datacenter=eqiad/year=2024/month=1/day=17"
spark3-sql -e "show partitions event.mediawiki_page_move;" | grep "datacenter=eqiad/year=2024/month=1/day=15"

# Rerun in Airflow at /dags/wikidata_item_page_link_weekly/grid

# check wiki page link dataset snapshots have been created
spark3-sql -e "show partitions wmf.wikidata_item_page_link;"
#partition
#snapshot=2023-12-04
#snapshot=2023-12-11
#snapshot=2023-12-18
#snapshot=2023-12-25
#snapshot=2024-01-01
#snapshot=2024-01-08
#snapshot=2024-01-15
#snapshot=2024-01-22
#Time taken: 0.123 seconds, Fetched 8 row(s)

Antoine_Quhen updated the task description. (Show Details)Jan 30 2024, 11:12 AM

Antoine_Quhen updated Other Assignee, added: Antoine_Quhen.

• lbowmaker edited projects, added: Data-Engineering (Sprint 8); removed: Data-Engineering (Sprint 7).Jan 30 2024, 12:22 PM

Antoine_Quhen moved this task from Next Up to Done on the Data-Engineering (Sprint 8) board.Jan 31 2024, 12:56 PM

wmf.wikidata_item_page_link/snapshot=2024-01-15 looked fine from a row count perspective but 2024-01-22 snapshot was unexpectedly available and had zero rows:

select count(*) from wmf.wikidata_item_page_link where snapshot='2024-01-22';
...
+------+
| _c0  |
+------+
| 0    |
+------+

The empty partition was created when I resolved the previous job. I've marked the 1/22 run as succeeded in order to stop it and make room for the other run execution. That was a mistake, and after catching up on 1/15, it supposed that 1 of the source wikidata_entity was present for 1/22. And that was not the case, leading to an empty wikidata_item_page_link

Now, we have removed the partition 1/22 and the Airflow run is now waiting for wikidata_entity.

mfossati updated the task description. (Show Details)Jan 31 2024, 6:12 PM

@dcausse , analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15 is now available.

NOTE: we deployed a delta computation change, which is now done against discovery.cirrus_index_without_content, see T338013: [L] Create search index deltas by comparing to `discovery.cirrus_index_without_content` in hive. This is the first time the pipeline ships such a dataset: it has more rows than the past ones, i.e., 311,545 VS 130,903 from the previous snaphsot. This is expected, and detailed in T338013#9462502. Would you be so kind to double-check it before back-filling your DAG?

mfossati updated the task description. (Show Details)Feb 1 2024, 12:37 PM

@dcausse , this ticket is now unblocked. 2024-01-15 image suggestions are shipped, and 2024-01-22 ones are running. Please see my comment above.

@mfossati if 2024-01-22 is running it probably means that it's comparing against an index that do not have 2024-01-15 suggestions, should I skip 2024-01-15 and let the dag pick 2024-01-22?

In T356030#9509840, @dcausse wrote:

@mfossati if 2024-01-22 is running it probably means that it's comparing against an index that do not have 2024-01-15 suggestions, should I skip 2024-01-15 and let the dag pick 2024-01-22?

You're totally right, that's the point of having T338013: [L] Create search index deltas by comparing to `discovery.cirrus_index_without_content` in hive. I think you can safely skip 2024-01-15.

@dcausse please let me double-check everything first.

@dcausse it turned out that the analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-22 has zero rows. I speculate this is due to T356030#9502009. As a result, I've cleared the 2024-01-22 DAG and paused it so that you can pick 2024-01-15 up.
2024-01-15 looks good: please double-check it and pick it up.
Thanks for bearing with me!

@mfossati OK resumed the dag to process 2024-01-15, marked 2024-01-22 explicitly as failed while you figure out what's going on there.

dcausse updated the task description. (Show Details)Feb 2 2024, 5:33 PM

Gehel triaged this task as High priority.Feb 5 2024, 12:32 PM

Gehel moved this task from needs triage to Current work on the Discovery-Search board.

Gehel edited projects, added: Discovery-Search (Current work); removed: Discovery-Search.

Gehel moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Feb 5 2024, 4:35 PM

Gehel closed this task as Resolved.Feb 9 2024, 9:21 AM

Gehel claimed this task.

Search dag image_suggestions_weekly failed waiting for analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15Closed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Search dag image_suggestions_weekly failed waiting for analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15
Closed, ResolvedPublic
Actions