Table creation: https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/blob/main/automoderator/create_automoderator_config.hql

kcvelaga opened https://gitlab.wikimedia.org/repos/product-analytics/automoderator-metrics-jobs/-/merge_requests/1

Job script to update automoderator's latest config info

kcvelaga merged https://gitlab.wikimedia.org/repos/product-analytics/automoderator-metrics-jobs/-/merge_requests/1

Job script to update automoderator's latest config info

kcvelaga opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/836

DAG to fetch and process Automoderator's config, weekly.

jebe merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/836

DAG to fetch and process Automoderator's config, weekly.

Mentioned in SAL (#wikimedia-operations) [2024-09-20T14:03:57Z] <kcvelaga@deploy1003> Started deploy [airflow-dags/analytics_product@6010568]: automoderator config dag T375062

Mentioned in SAL (#wikimedia-operations) [2024-09-20T14:04:43Z] <kcvelaga@deploy1003> Finished deploy [airflow-dags/analytics_product@6010568]: automoderator config dag T375062 (duration: 01m 47s)

This pipeline is running end to end successfully. We can resolve after verfying the next run.

Analytics cluster

• Table creation: create_automoderator_config.hql
  • creates table at wmf_product.automoderator_config
• Job script: extract_automoderator_config.py
• Airflow DAG: automoderator_config_weekly_dag.py
  • scheduled to run weekly; updates the latest snapshot to https://analytics.wikimedia.org/published/datasets/automoderator/automoderator_config.tsv

Toolforge

• Table creation: create_automoderator_config_table.sql
  • create_automoderator_config_table.sh creates automoderator_config in s56213__unified_automod_metrics_p
• Update script: update_automoderator_config.py
  • fetches from the web published TSV and updates the database in ToolsDB
• Weekly job script: update_automoderator_config_weekly.sh

The snapshots of config are being successfully published on a weekly basis at https://analytics.wikimedia.org/published/datasets/automoderator/