Some years ago we got approval to update our query_clicks data processing pipeline and update our retention times to 13 months. This still needs to be implemented. For the most part this is adjusting the query_clicks hourly and daily airflow dags to follow the requirements set forth in the approval.
Related approval: https://phabricator.wikimedia.org/T235858
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| search: mjolnir: version bump mjolnir dep | repos/data-engineering/airflow-dags!1186 | gmodena | bump-mjolnir | main | |
| cli: refactor start/end date params. | repos/search-platform/mjolnir!14 | gmodena | refactor-date-params-to-cli | main | |
| search: mjolnir: fix query_clicks_ltr date argument order | repos/data-engineering/airflow-dags!1179 | gmodena | mjolnir-start-end-dates | main | |
| fix: pass start_date and end_date to query_clicks.transformer | repos/search-platform/mjolnir!13 | gmodena | query_ltr_param | main | |
| search: query_clicks: increase data retention | repos/data-engineering/airflow-dags!1105 | gmodena | clickstream-increase-retention | main |
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1105
Draft: search: query_clicks: hash session ids
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1105
search: query_clicks: increase data retention
gmodena opened https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/13
fix: pass start_date and end_date to query_clicks.transformer
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1179
search: mjolnir: fix query_clicks_ltr date argument order
ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/13
fix: pass start_date and end_date to query_clicks.transformer
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1179
search: mjolnir: fix query_clicks_ltr date argument order
gmodena opened https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/14
Draft: cli: refactor start/end date params.
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1186
Draft: search: mjolnir: version bump mjolnir dep
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1186
search: mjolnir: version bump mjolnir dep