Background:
Wikidata has always been a target for search engine optimization but it seems to be increasing significantly recently. People seem to be led to Wikidata as the place to anchor their entity to be accessible in generative AI answers etc. To help ensure promotional editing is being spotted and properly acted upon, a revert risk model has been created. The analytics team would like to support this work and provide metric monitoring for the Revert Risk Model usage and impact to this overall topic.
Questions & Metrics: Revert Risk Model
Question: Is the new risk model successful: are bad edits being reverted?
- %/# of bad edits that get reverted.
- % of bad edits that are reverted within 2 days.
Bad Edit Definition:
As per the work done in T429049, we can use the table event.mediawiki_page_revert_risk_prediction_change_v1 and the value from predicted_classification.probabilities['true'], below shows the ranges to choose for the model(s) and classifications of a bad edit:
| Tier | Damaging (old) | RR floor 0.641 | RR floor 0.371 |
| likelygood range | [<0.279] | [<0.650] | [<0.371] |
| likelybad range | [0.385] | [0.650] | [0.371] |
| likelyverybad range | [0.929] | [0.889] | [0.889] |
Further example code here, in order to review and incorporate bot edits, correct revision IDs, etc.
Core source tables:
- event.mediawiki_page_change_v1
- event.mediawiki_page_revert_risk_prediction_change_v1
- Join on page / revision IDs
**Deliverables**
- Task code for bad edits and metrics
- Snapshot of data exported
- Superset dashboard
Data Snapshot Creation
- Define metric(s) to be derived from the new / updated data pipeline
- Define output table columns with stakeholder(s)
- Create process diagrams in query documentation to map out complex flows
- Write job queries or code
- Test job queries or code
- Export data snaphsot / add to wmde hdfs
- Create Superset dashboard
Previous notes from initial session: There are some notes / decisions to be made for the above metrics: # We need to firm up a definition of “bad edits”. See above added section for this definition based on T429049. # For these metrics we should include the option of filtering on item notability. Answer - we actually don't combine this as they are quite different. # Can we have an overview of training data? Not necessary, instead what is more useful are the findings from ticket T429049