Goal
Migrate admin_request_month to airflow. Currently the staging table is queried from mariadb cu_log. After T364398 is completed, we are going to query the data from hive table and enable data pipeline in airflow.
Description
Description
Details
Details
- Other Assignee
- • JEbe-WMF
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jwang | T364406 Migrate IP masking dashboard data pipeline to airflow | |||
| Resolved | jwang | T380318 Migrate admin_request_month schema to airflow |
Event Timeline
Comment Actions
The queries files have been checked in through Gitlab merge, changes include:
- One file to create the destination table trust_safety_admin_request_monthly
- One file to pull and ingest data into trust_safety_admin_request_monthly. It pulls the most recent one month's data once the latest snapshot is available.
@JEbe-WMF, pass the ticket on to you for work related to the DAG files.
Comment Actions
@JEbe-WMF has checked in and merged the DAG file. Thank you for your work.
Merge requests are:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/968
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1060
Comment Actions
I have backfilled the historical data from jiawang2.admin_request_latest_month to wmf_product.trust_safety_admin_request_monthly.
Also updated the source table of Check User Requests in IP masking dashboard to use the new schema.