This will include multiple pipelines, with seperate sub-tasks as needed.

• The first set of pipelines will be using Airflow to calculate the required metrics, and publishing to https://analytics.wikimedia.org/published/datasets
• The second set of pipelines will be using Toolforge jobs framework to read from the published dataset, and update the db in ToolsDB (which will be used by the dashboard).

Set A (Airflow jobs)

To aggregate and published metrics.

Automoderator config

• Data source: https://noc.wikimedia.org/conf/InitialiseSettings.php.txt
  • wmgUseAutoModerator & wgAutoModeratorUsername
• Processing: Job repo & Airflow DAG
• Output:
  • wmf_product.automoderator_config
  • https://analytics.wikimedia.org/published/datasets
• Frequency: weekly

Daily monitoring metrics

• Data source: MariaDB-replicas (tables: revision, change_tag, change_tag_def, actor, user)
• Processing: SQL queries (tweaked version of automod_reverts_info.sql can be used), Job script and Airflow DAG
• Output:
  • wmf_product.automoderator_monitoring_metrics_daily
  • https://analytics.wikimedia.org/published/datasets
• Frequency: daily

Monthly key metrics

• Data source: wmf.mediawiki_history
• Processing: HQL queries and Airflow DAG
• Output:
  • wmf_product.automoderator_key_metrics_monthly
  • https://analytics.wikimedia.org/published/datasets
• Frequency: monthly

Revert proportion handled monthly

• Data source: wmf.mediawiki_history
• Processing: HQL queries and Airflow DAG
• Output:
  • wmf_product.automoderator_revert_proportion_monthly
  • https://analytics.wikimedia.org/published/datasets
• Frequency: monthly

Set B (Toolforge jobs)

To load the published metrics into ToolsDB.

• A daily job to update the monitoring metrics.
• A monthly job to update the key metrics.
  • maybe split into another one to update the revert proportion handled data.