Mjolnir has been failing feature collection for several weeks in a row now. The most recent run finishes with:
Exception: Did not collect equal number of rows per feature
Figure out what happened to feature collection and get everything running again.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| search: query_clicks: Parameterize max_active_runs for backfills | repos/data-engineering/airflow-dags!2078 | ebernhardson | work/ebernhardson/query-clicks-parameterize | main | |
| search: query-clicks: Repair null timestamps | repos/data-engineering/airflow-dags!2075 | ebernhardson | work/ebernhardson/query-clicks-null-timestamp | main |
Had a bit of time to start looking into this, some findings:
Not sure yet what changed, but we have a direction to look in. Not clear what we can do about all the missing session_id's at this point, likely they can't be reconstructed as we don't retain enough data.
ebernhardson opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2075
search: query-clicks: Repair null timestamps
dcausse merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2075
search: query-clicks: Repair null timestamps
ebernhardson opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2078
search: query_clicks: Parameterize max_active_runs for backfills
ebernhardson merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2078
search: query_clicks: Parameterize max_active_runs for backfills
Problem was traced down to null timestamps coming out of query_clicks_hourly. This was due to an overly specific format specifier and the source data adding millisecond precision to the timestamp. Timestamp conversion was changed to a more permissive conversion. The last three months of query_clicks_hourly and query_clicks_daily were backfilled. mjolnir dag was unpaused and completed a run.
I was curious about changes to the training loss, so I put together a graph of the final ndcg achieved for each weekly run since 2018. For the most part it shows training is mostly back to historical norms.
@EBernhardson / @pfischer: The only associated project tag has been archived. If this task is done, please change its status to resolved; if it is not done, please associate an active project tag. Thanks!