Background
We want to understand the impact of temporary accounts rollout on our projects. This is a significant change and we want to be careful about how temporary accounts impact the ecosystem. This is especially true now with upcoming pilot wiki deployments in the near future.
Request
Metrics brainstormed together between Product, Design Research and Comms:
- Edits rollbacks T371404, T377516
- Revert rate of temp account edits
-
Number of temp accounts being created
Note: drop it from this ticket. as the source table is currently only available in mariadb. Discussed it with engineer, it will be monitored in Grafana instead (T375505)
- Number of temp accounts being blocked versus IP addresses T376080, T377516
-
(Edit) Traffic IPs versus temp accounts: number edits by IP. & number of edits by temp accounts
Note: deferred, Reason: T372481#10119239
-
New checkuser admins
Note: Dropped from this request. Reason: T372481#10119239. Propose measuring number of checkuser admins instead. - New regular account creations T377516
-
Number of IP reveals: Need to request access to the logging table
Note: Dropped from this request. Reason: T372481#10119239 -
Number of rate limit trips for temporary account creations per IP (default is 6 temp account creations per IP address per day).
Note: Dropped from this request and will be tracked under another task based on the discussion in T371402#10116224 - NEW: number of checkuser admins T377516
List will change based on what is feasible to include. The list should also be reviewed and the order/priority of the metrics decided.
Timeline
For Temp Accounts-related metrics, they should be in place by the time Temp Accounts is deployed to pilot wikis and ready to be measured.
For non-Temp Accounts-related metrics, we should have measurements by October (or November at the latest) to have baselines available before deployment.
Acceptance Criteria
- Documentation of metric definitions and how we will measure them or flagging that a metric is not measurable
- Development of queries
- Data QA
- Airflow-based pipelines (from the start so we're not adding to T364406)
- Queries added to https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/tree/main/trust_safety_metrics
- DAGs added to https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/tree/main/analytics_product/dags/trust_safety_metrics
- DAGs deployed to analytics-product Airflow instance
- Backfill measurements where possible
- Add to IP Masking dashboard in Superset or temp account deployment dashboard