We have improved the readability model a lot (using a ranking instead of classification approach, see TRank vs LMC in the paper's results). Currently, the model on LiftWing still uses the older LMC (classification). If possible, we would like to replace it with the new TRank model.
See below for more details following the instructions for requesting an update.
Which model needs updating?
We would like to update the Readability model: https://api.wikimedia.org/wiki/Lift_Wing_API/Reference/Readability_score_object and https://api.wikimedia.org/wiki/Lift_Wing_API/Reference/Get_readability_prediction
What changes have been made to the model? (e.g., updated training data, different approach, new features, etc.)
The model was trained using a different approach (ranking instead of classification). The FK-score approximation model was changed to a simple linear transformation instead of the nonlinear regression model.
Do any dependent repositories/packages need updates? (e.g., knowledge integrity, sklearn, pytorch, etc.) Please provide the MR/version for reference.
Package readability-liftwing should be updated according to the MR: https://gitlab.wikimedia.org/trokhymovych/readability-liftwing/-/merge_requests/6
Is there a new model binary? What is its version?
Yes. Its version is 4 (the same as mentioned in MR). The new model binary can be found here: https://drive.google.com/file/d/1wsmx5nw2_EtrRlA2RfDXBEiO-SivPYcU/view?usp=sharing .
Does the input/output schema need any changes?
The output schema needs to be changed. The new schema is represented as
@dataclass class ReadabilityResult: score: float fk_score_proxy: float
where score stands for the readability score provided by the ranking model (can be used to compare articles between each other), and fk_score_proxy is a Flesch–Kincaid score approximation. The schema is different because the model is now formulated as a ranking task (only assigning a score) instead of a binary prediction task (True/False with probability score for each class).
Does the preprocessing stage require changes?
Does the prediction stage require changes?
The preprocessing and prediction stages have minor changes (due to model change), which are represented in the corresponding MR.
Checklist:
- Update the model card
- The model card is updated and can be found here: https://meta.wikimedia.org/wiki/Machine_learning_models/Proposed/Multilingual_readability_model_card
- Provide the location of the new model binary and its sha512 to the ML team.
- The new model binary can be found here: https://drive.google.com/file/d/1wsmx5nw2_EtrRlA2RfDXBEiO-SivPYcU/view?usp=sharing .
- sha512: d1bbf9173091b45a8940f14cd4b3b113374d85c88b5bc5e09c2f6d5676084013cff91a65c1da161c2e970274a0a80b2392b3cc02c40f385602786562fd5a5d3f
Additional note: The set of supported languages changed slightly because we use a different base model (see model card)
'af', 'sq', 'am', 'ar', 'hy', 'as', 'az', 'eu', 'be', 'bn', 'bs', 'br', 'bg', 'my', 'ca', 'zh-yue', 'zh', 'zh-classical', 'hr', 'cs', 'da', 'nl', 'en', 'eo', 'et', 'tl', 'fi', 'fr', 'gl', 'ka', 'de', 'el', 'gu', 'ha', 'he', 'hi', 'hu', 'is', 'id', 'ga', 'it', 'ja', 'jv', 'kn', 'kk', 'km', 'ko', 'ku', 'ky', 'lo', 'la', 'lv', 'lt', 'mk', 'mg', 'ms', 'ml', 'mr', 'mn', 'ne', 'no', 'or', 'om', 'ps', 'fa', 'pl', 'pt', 'pa', 'ro', 'ru', 'sa', 'gd', 'sr', 'sd', 'si', 'sk', 'sl', 'so', 'es', 'su', 'sw', 'sv', 'ta', 'te', 'th', 'tr', 'uk', 'ur', 'ug', 'uz', 'vi', 'cy', 'fy', 'xh', 'yi', 'simple'