Page MenuHomePhabricator

Mjolnir feature collection failing in mjolnir_weekly Airflow DAG
Open, LowPublic5 Estimated Story Points

Description

Mjolnir has been failing feature collection for several weeks in a row now. The most recent run finishes with:

Exception: Did not collect equal number of rows per feature

Figure out what happened to feature collection and get everything running again.

Event Timeline

pfischer set the point value for this task to 5.Jan 26 2026, 4:36 PM
dr0ptp4kt renamed this task from Mjolnir feature collection failing to Mjolnir feature collection failing in mjolnir_weekly Airflow DAG.Wed, Mar 4, 4:28 PM

Had a bit of time to start looking into this, some findings:

  • Feature collection fails because the input querys are empty
  • Input queries are empty because query_clicks_ltr filters by session count, but all the session_id's are null
  • The first partition that is missing session_id's is discovery.query_clicks_daily/year=2025/month=8/day=28
  • Those partitions are generated by an hql query, the query and the airflow dag that issues it hasn't been changed since may 2025.

Not sure yet what changed, but we have a direction to look in. Not clear what we can do about all the missing session_id's at this point, likely they can't be reconstructed as we don't retain enough data.