Initial baseline analysis for reference: https://gitlab.wikimedia.org/kcvelaga/automoderator-measurement/-/blob/main/baselines/T348863_content_moderation_backlogs_flagged_revs.ipynb.ipynb?ref_type=heads
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Queries and notebooks for moderation metrics (related to Automoderator) | repos/product-analytics/data-pipelines!13 | kcvelaga | automoderator_puppet | main | |
| Draft: dummy: review fr_pending script | kcvelaga/pyspark-conda-mariadb!1 | kcvelaga | dummy_mr | main | |
| fr_pending_hourly | repos/product-analytics/data-pipelines!12 | kcvelaga | fr_pending | main | |
| fr_pending_hourly | repos/product-analytics/data-pipelines!11 | kcvelaga | fr_pending | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | KCVelaga_WMF | T362612 Create ETL pipelines for Automoderator baseline metrics | |||
| Resolved | KCVelaga_WMF | T362615 ETL pipeline for flaggedrevs metrics (pending frevs hourly) |
Adding a note for future reference, that this pipeline requires looping through a list of wikis, some example DAGs for reference:
kcvelaga opened https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/11
fr_pending_hourly
kcvelaga closed https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/11
fr_pending_hourly
kcvelaga opened https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/12
fr_pending_hourly
kcvelaga updated https://gitlab.wikimedia.org/kcvelaga/pyspark-conda-mariadb/-/merge_requests/1
Draft: dummy: review fr_pending script
kcvelaga closed https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/12
fr_pending_hourly
Gerrit bot has not commented!
Patch for review at : https://gerrit.wikimedia.org/r/c/analytics/wmf-product/jobs/+/1049106
kcvelaga opened https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/13
Queries and notebooks for moderation metrics (related to Automoderator)
bearloga merged https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/13
Queries and notebooks for moderation metrics (related to Automoderator)
kcvelaga closed https://gitlab.wikimedia.org/kcvelaga/pyspark-conda-mariadb/-/merge_requests/1
Draft: dummy: review fr_pending script
This DAG is finally working! Code related to the pipeline
Data is available at: wmf_product.moderation_flagged_revisions_pending_hourly
I will check the data in a week again, and resolve if there are no issues.
Mentioned in SAL (#wikimedia-operations) [2025-03-13T14:00:43Z] <kcvelaga@deploy2002> Started deploy [airflow-dags/analytics_product@554407c]: T362615
Mentioned in SAL (#wikimedia-operations) [2025-03-13T14:01:18Z] <kcvelaga@deploy2002> Finished deploy [airflow-dags/analytics_product@554407c]: T362615 (duration: 01m 39s)