[Analytics] [Bug] wd_item_sitelink_segments_weekly sensor isn't firing
Open, Needs TriagePublic
Actions

Assigned To

Authored By

	AndrewTavis_WMDE
	Feb 13 2025, 11:01 AM

Description

Wikidata Analytics Bug Report

This task was generated using the Wikidata Analytics bug report form. Please use the task template linked on our project page to report bugs to the team. Thank you!

Behavior

Please provide a concise description of what you’re experiencing and what you’d expect to happen.

Currently the wd_item_sitelink_segments_weekly DAG isn't finishing because the sensor isn't completing. The sensor for this DAG is:

sensor = dataset("hive_wmf_wikidata_entity").get_sensor_for(dag)

Ultimately the error that we're getting from this is that the sensor has timed out. This could be an upstream issue with hive_wmf_wikidata_entity.

Deadline

Please make the time sensitivity of this bug report clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority.

DD.MM.YYYY

Information below this point is filled out by the task assignee.

Assignee Planning

Sub Tasks

A full breakdown of the steps to complete this task.

Derive what's causing the sensor to not fire (Snapshot was missing - see T386255)
Check if there are other changes that need to happen to the DAG
- Is there a new dataset, and do fields need to change?
- Was just that the underlying data was missing
Make any needed changes to the DAG
- No changes needed, and we'll have a month of data loss
Reduce the SLA timedelta of the DAG as the alerts we're getting from it are likely because of a buildup of weekly DAGs (see T382878)
- Included in airflow-dags#1256
Redeploy/unpause DAG

Estimation

Estimate: 1.5 days
Actual: 30 minutes (nothing to really do except document, restart and check)

Notes

Things that came up during the completion of this task, questions to be answered and follow up tasks.

Note

Related Objects

Mentioned In: T382878: [Analytics] [Bug] WMDE Airflow DAGs are sending excessive SLA config requirement emails
Mentioned Here: T386255: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20
T382878: [Analytics] [Bug] WMDE Airflow DAGs are sending excessive SLA config requirement emails

Event Timeline

AndrewTavis_WMDE created this task.Feb 13 2025, 11:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2025, 11:01 AM

AndrewTavis_WMDE moved this task from Incoming to Kanban on the Wikidata Analytics board.Feb 13 2025, 11:01 AM

AndrewTavis_WMDE edited projects, added Wikidata Analytics (Kanban); removed Wikidata Analytics.

AndrewTavis_WMDE claimed this task.Mar 12 2025, 9:32 AM

AndrewTavis_WMDE mentioned this in T382878: [Analytics] [Bug] WMDE Airflow DAGs are sending excessive SLA config requirement emails.Tue, Apr 1, 9:44 AM

AndrewTavis_WMDE updated the task description. (Show Details)

Moving to To-Do as a reminder to myself as we're at 47 days since this issue was made, meaning we're creeping towards data loss territory.

AndrewTavis_WMDE moved this task from To-Do to In Progress on the Wikidata Analytics (Kanban) board.Fri, Apr 25, 1:55 PM

Moved to In Progress because the data retention deadline is coming, but it seems like the data in question isn't there anyway. The dump in question to create wmf.wikidata_entity was stuck at the 2025-01-20 snapshot and only restarted at 2025-03-03. See T386255: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20. I'll restart the DAG at March 2025 and mark all runs in February as failed given there is no data to generate the metrics.