Improve the current quality score metric by:
> 0.36 (Stub+); > 0.54 (C+) > 0.65 (B+) > 0.78 (GA+) > 0.88 (FA)
Thresholds obtained as follows: the upper limit of each quality class is the median predicted quality score of revisions corresponding to such class as labeled by editors.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Miriam | T331154 The Knowledge Gap Index | |||
| Open | Miriam | T331155 Knowledge Gaps Metrics | |||
| Resolved | fkaelin | T331161 Integrate Revision Quality Metric | |||
| Resolved | fkaelin | T332383 Modify current quality metrics |
FYI, I computed the monthly median quality of the last revision available of each article (I mean "available" because many articles might not have been updated in a given month). Data is stored at riskobservatory.predicted_quality_last_revisions and exposed in the v2 of the Risk Observatory https://superset.wikimedia.org/superset/dashboard/p/GLzB1ZorMdy (I also computed the yearly quartiles to expose this data with boxplots https://superset.wikimedia.org/superset/dashboard/p/3j5vK0DBYzl)
@Fabian let's put this on hold as I learnt that scores are comparable across languages. We should probably think of a way of getting thresholds relative to each language.
Updates after a chat with @Isaac:
From @Isaac too: Finally we could do something like above but instead of trying to choose a quality class and explain that, just use the model to set basic standards around the features we care about -- e.g., # of articles with at least 5 references, 2 sections, an infobox, an image, and 3 categories, or something like that. that's way more transparent.
@Fabian, with the help of @Pablo and @Isaac, we have defined a new set of rules to create a new "global quality score" that reflects a minimum acceptable quality across Wikipedias.
We look at the association between page properties (length, number of images, references, etc.) and community-assigned quality labels from 5 wikis – hungarian, turkish, english, french and arabic. Our initial recommendation is that an article is of acceptable quality if it meets at least 5 of the 6 following criteria:
In practice, part of the code to calculate this is in here, in the section "Relaxing Constraints": https://public-paws.wmcloud.org/User:Miriam_(WMF)/analyze_standard_quality.ipynb (allsample is the dataframe with quality features from all wikis).
The final metric would be the % of articles that meet at least 5 or the 6 criteria above.
How long do you think would it take to compute this new metric based on existing features? If this is generally feasible, I will change the description of the task with this new metric details.
Created an initial implementation of a 'standard quality' as feature metric of the knowledge gap pipelines (MR)
The standard quality flag is available for all historical revisions, e.g. for the article on the Arc de triomphe on fr wiki:
The MR also takes a first stab at forward filling missing data for quality scores, but this is more subtle than expected. For distributed compute we prefer to operate on sparse data, e.g. if there is no revision to a given article in a given month, instead of counting a 0 we prefer to simply leave that datapoint out. For the reader/contributor based metrics that makes sense since it measures activity, however for content based metrics like quality scores we want to know about the state of an article in a given month even if there has been no other events that would "cause" that article to appear in the data. The forward fill implemented in the MR is not sufficient as we can't forward fill a missing article score (a NaN value) if the row itself is not present in the dataset. Due to the nature of the nested aggregation done for content gaps pipeline (the metrics feature are aggregated, then re-aggregated for each content gap) this means we need to make the metric features non-sparse as a whole. I will implement/test this next week.