Page MenuHomePhabricator
Create Task
Maniphest T362615

ETL pipeline for flaggedrevs metrics (pending frevs hourly)
Closed, ResolvedPublic
Actions

Description

  • Revisions to be reviewed
  • Time elapsed since revision
  • Median time to be reviewed
  • Unique reviewers
  • Average # Reviews per reviewer

Initial baseline analysis for reference: https://gitlab.wikimedia.org/kcvelaga/automoderator-measurement/-/blob/main/baselines/T348863_content_moderation_backlogs_flagged_revs.ipynb.ipynb?ref_type=heads

Related Objects
Search...

Event Timeline

KCVelaga_WMF triaged this task as Medium priority.Apr 16 2024, 8:36 AM
KCVelaga_WMF created this task.
KCVelaga_WMF edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
KCVelaga_WMF moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.
Samwalton9-WMF moved this task from Backlog to Data/Research on the Automoderator board.Apr 17 2024, 8:59 AM
Samwalton9-WMF edited projects, added Moderator-Tools-Team (Kanban); removed Moderator-Tools-Team.Apr 23 2024, 12:24 PM
Samwalton9-WMF moved this task from Ready to In Progress on the Moderator-Tools-Team (Kanban) board.
KCVelaga_WMF added a comment.May 20 2024, 5:56 PM

Adding a note for future reference, that this pipeline requires looping through a list of wikis, some example DAGs for reference:

CodeReviewBot added a project: Patch-For-Review.Jun 4 2024, 4:07 PM

kcvelaga opened https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/11

fr_pending_hourly

CodeReviewBot added a comment.Jun 5 2024, 4:35 PM

kcvelaga closed https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/11

fr_pending_hourly

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2024, 5:31 PM
CodeReviewBot added a project: Patch-For-Review.Jun 6 2024, 7:09 AM

kcvelaga opened https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/12

fr_pending_hourly

KCVelaga_WMF moved this task from In Progress to Eng review on the Moderator-Tools-Team (Kanban) board.Jun 7 2024, 7:39 AM
KCVelaga_WMF renamed this task from ETL pipeline for flaggedrevs metrics to ETL pipeline for flaggedrevs metrics (pending frevs hourly).Jun 10 2024, 5:23 AM
KCVelaga_WMF changed the task status from Open to In Progress.
KCVelaga_WMF mentioned this in T366869: Add cx_translators to sqoop.Jun 11 2024, 9:10 AM
CodeReviewBot added a comment.Jun 14 2024, 11:15 AM

kcvelaga updated https://gitlab.wikimedia.org/kcvelaga/pyspark-conda-mariadb/-/merge_requests/1

Draft: dummy: review fr_pending script

KCVelaga_WMF mentioned this in T367848: Publishing conda environments with WMF Data Workflow Utils is broken.Jun 18 2024, 5:44 AM
CodeReviewBot added a comment.Jun 18 2024, 9:59 AM

kcvelaga opened https://gitlab.wikimedia.org/kcvelaga/automoderator-baseline-pipelines-mariadb/-/merge_requests/1

Resolve T362615

CodeReviewBot added a comment.Jun 18 2024, 10:00 AM

kcvelaga closed https://gitlab.wikimedia.org/kcvelaga/automoderator-baseline-pipelines-mariadb/-/merge_requests/1

Resolve T362615

CodeReviewBot added a comment.Jun 25 2024, 5:04 AM

kcvelaga closed https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/12

fr_pending_hourly

KCVelaga_WMF added a comment.Jun 25 2024, 5:06 AM

Gerrit bot has not commented!

Patch for review at : https://gerrit.wikimedia.org/r/c/analytics/wmf-product/jobs/+/1049106

CodeReviewBot added a comment.Jul 2 2024, 6:24 AM

kcvelaga opened https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/13

Queries and notebooks for moderation metrics (related to Automoderator)

CodeReviewBot added a comment.Jul 16 2024, 1:25 PM

bearloga merged https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/13

Queries and notebooks for moderation metrics (related to Automoderator)

CodeReviewBot added a comment.Aug 8 2024, 10:34 AM

kcvelaga closed https://gitlab.wikimedia.org/kcvelaga/pyspark-conda-mariadb/-/merge_requests/1

Draft: dummy: review fr_pending script

KCVelaga_WMF moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.Aug 30 2024, 5:20 PM

This DAG is finally working! Code related to the pipeline

Data is available at: wmf_product.moderation_flagged_revisions_pending_hourly

I will check the data in a week again, and resolve if there are no issues.

KCVelaga_WMF removed a project: Patch-For-Review.Aug 30 2024, 5:21 PM
AndrewTavis_WMDE mentioned this in T360296: [Analytics] Implement data process to identify missing Wiktionary entries .Sep 5 2024, 12:43 PM
Iflorez mentioned this in T365404: Create a plan for moving campaigns-product metrics data to airflow.Sep 10 2024, 7:02 PM
Iflorez mentioned this in T374500: ETL pipeline for campaigns-product key metrics: Destination tables schemas.Sep 10 2024, 11:07 PM
Iflorez mentioned this in T374491: Create ETL pipelines for campaigns-product baseline metrics.Sep 10 2024, 11:10 PM
KCVelaga_WMF closed this task as Resolved.Sep 11 2024, 1:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits