Page MenuHomePhabricator
Create Task
Maniphest T364406

Migrate IP masking dashboard data pipeline to airflow
Closed, ResolvedPublic
Actions

Description

Goal

The IP Masking (Temp Accounts) guardrail metrics dashboard currently is maintained by the product analytics team. The staging tables are fetched monthly by a cron job into a database under jwang's account. We plan to migrate the queries to Airflow, so that it can be fully automated and easily maintained.

Dependency

Add cu_log to the data lake. T364398

To Do

  • Write updated SQL queries for table creation and updating
  • Write Airflow DAG
  • Build new version of tables
  • Backfill old data to new tables T378518
  • Update metric queries to use new tables
  • Update Superset dashboard to use new tables

Related Objects
Search...

Event Timeline

jwang created this task.May 7 2024, 5:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2024, 5:00 PM
mpopov mentioned this in T364398: Add MW table 'cu_log' to data lake.May 14 2024, 4:12 PM
kostajh added a project: Temporary accounts.May 21 2024, 1:21 PM
kostajh subscribed.
jwang moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.May 29 2024, 5:24 PM
Tchanders moved this task from Inbox to Figure out Analytics/Instrumentation for Temp Accounts rollout on the Temporary accounts board.Jun 4 2024, 12:58 PM
Tchanders edited projects, added: Temporary accounts (Figure out Analytics/Instrumentation for Temp Accounts rollout); removed: Temporary accounts.
jwang added a comment.Jun 5 2024, 5:31 PM
  • Write updated SQL queries for table creation and updating

Queries have been checked in to Gitlab: link

jwang updated the task description. (Show Details)Jun 5 2024, 5:32 PM
jwang added a comment.EditedJun 5 2024, 8:59 PM

DAG files for trust_safety_admin_action_daily have been submitted to gitlab. A request for review and merge has been made. link

mpopov triaged this task as Medium priority.Jun 11 2024, 1:00 PM
mpopov changed the task status from Open to Stalled.Jul 30 2024, 2:44 PM
mpopov mentioned this in T371402: [WE4.4] Additional monitoring capabilities for rollout of Temporary Account.
mpopov moved this task from Doing to Blocked on the Product-Analytics (Kanban) board.

Pausing this work in favor of T371402: [WE4.4] Additional monitoring capabilities for rollout of Temporary Account

mpopov mentioned this in T346466: Investigate how temporary accounts are logged and categorized.Jul 30 2024, 3:18 PM
CodeReviewBot added a project: Patch-For-Review.Aug 8 2024, 7:59 PM

bearloga opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/795

analytics_product: T&S admin actions: fix MW sensor

CodeReviewBot added a comment.Aug 13 2024, 3:10 PM

bearloga merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/795

analytics_product: T&S admin actions: fix MW sensor

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2024, 3:30 PM
Mayakp.wiki subscribed.Sep 9 2024, 7:02 PM
jwang updated the task description. (Show Details)Oct 18 2024, 9:30 PM
jwang added a comment.Oct 18 2024, 9:36 PM

DAG files for following tables have been checked-in to gitlab. Here is the merge request, which is pending review. link

  • trust_safety_admin_action_monthly
  • trust_safety_block_monthly
  • trust_safety_admin_monthly
  • trust_safety_new_admin_monthly
jwang added a comment.Oct 29 2024, 4:43 PM
Status update

Completed Tasks
The following pipelines have been deployed. The data from the recent snapshot (2024-09) has been backfilled

  • trust_safety_admin_action_monthly
  • trust_safety_block_monthly
  • trust_safety_admin_monthly
  • trust_safety_new_admin_monthly

Next steps

  • Backfill the historical data for the following tables
    • trust_safety_admin_action_monthly
    • trust_safety_block_monthly
    • trust_safety_admin_monthly
    • trust_safety_new_admin_monthly
  • Begin migrating admin_request_month table once the dependency (T364398) is resolved.
jwang updated the task description. (Show Details)Oct 29 2024, 4:48 PM
jwang changed the task status from Stalled to Open.Oct 29 2024, 4:50 PM
jwang added a comment.Dec 6 2024, 8:03 PM
Status update

Completed Tasks

  • Backfilled the historical data for the following tables
    • trust_safety_admin_action_monthly
    • trust_safety_block_monthly
    • trust_safety_admin_monthly
    • trust_safety_new_admin_monthly
  • Updated Superset dashboard to use the above new tables.

Next steps
Update Superset dashboard to use the new admin_request_monthly table after the migration (T380318) is done.

jwang closed subtask T378518: Backfill staging data from 4 old tables to new tables as Resolved.Dec 12 2024, 6:36 PM
jwang closed subtask T376081: Measure number of blocks on temp accounts as Resolved.Dec 20 2024, 7:15 PM
jwang closed subtask T380318: Migrate admin_request_month schema to airflow as Resolved.Feb 10 2025, 11:01 PM
jwang closed this task as Resolved.Feb 15 2025, 12:50 AM
jwang updated the task description. (Show Details)
• nettrom_WMF mentioned this in T333221: Product Analytics ETL Migration: IP masking metrics.Jun 10 2025, 12:05 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits