Page MenuHomePhabricator

Improving error message for Revertrisk models
Closed, ResolvedPublic3 Estimated Story Points

Description

Revertrisk models require a valid parent revision in order to measure the difference in quality and make a prediction. Therefore, they cannot be applied to (1) the first revision of a page or to (2) a revision that is the only one for a page.

Now the generic error message the model returns is unclear to users, and so they reported bugs (T351021, T347136). To address this, we should provide a more clear error message.

If we want to improve the error message for this case, it would probably require changing the code in the knowledge_integrity module, specifically, the revision.py. The get_parent_revision() will need to raise an error for parentless revisions, and then the model server can catch this error and return users a dedicated error message.

Note that the model server can't differentiate between a parentless revision (first revision), a missing page (page deletion), or missing user information, because in all these cases, the code returns a None back to the model server, which probably should be changed as well.

Event Timeline

The data models for revertrisk in knowledge_integrity are likely to be changed for work on task T352987. We will review this ticket afterwards.

achou raised the priority of this task from Medium to Needs Triage.Jan 18 2024, 9:57 AM
achou triaged this task as Medium priority.
achou raised the priority of this task from Medium to Needs Triage.
achou moved this task from Backlog/Lift Wing to Ready To Go on the Machine-Learning-Team board.
achou moved this task from Backlog/Lift Wing to Ready To Go on the Machine-Learning-Team board.
achou set the point value for this task to 3.

Knowledge Integrity v0.6.0 improved error representations by introducing an Error data class and different error codes for various situations when fetching MediaWiki API for revisions. (See https://gitlab.wikimedia.org/repos/research/knowledge_integrity/-/blob/main/knowledge_integrity/mediawiki.py?ref_type=heads#L14-26)

These changes enable us to provide users with specific error messages. However, we need to wait for pydantic v2, which Knowledge Integrity v0.6.0 uses, to be supported in KServe, so we can conduct proper testing for this.

Change 1011305 had a related patch set uploaded (by AikoChou; author: AikoChou):

[machinelearning/liftwing/inference-services@main] revertrisk: improve error messages

https://gerrit.wikimedia.org/r/1011305

Change #1011305 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] revertrisk: improve error messages

https://gerrit.wikimedia.org/r/1011305

Change #1014545 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] ml-services: update revertrisk-language-agnostic image

https://gerrit.wikimedia.org/r/1014545

This task is complete. Check out these examples of new error messages:

$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/revertrisk-language-agnostic:predict" -d '{"rev_id": 15925124, "lang": "ro"}' -H "Host: revertrisk-language-agnostic.revertrisk.wikimedia.org" --http1.1 -k |  jq '.'
{
  "detail": "Could not make prediction for revision 15925124 (ro). Reason: revision_missing"
}

$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/revertrisk-language-agnostic:predict" -d '{"rev_id": 1, "lang": "ro"}' -H "Host: revertrisk-language-agnostic.revertrisk.wikimedia.org" --http1.1 -k |  jq '.'
{
  "detail": "Could not make prediction for revision 1 (ro). Reason: parent_revision_missing"
}