Page MenuHomePhabricator

Align ORES prediction output with Lift Wing's one (for revscoring models)
Closed, ResolvedPublic

Description

The output of revscoring models on Lift Wing is different from the ORES ones, and this may end up being a problem for people migrating from one tool to the other one. For example, ad-hoc code has probably been written to get the score value from the following ORES response:

{
  "enwiki": {
    "models": {
      "goodfaith": {
        "version": "0.5.1"
      }
    },
    "scores": {
      "123456": {
        "goodfaith": {
          "score": {
            "prediction": true,
            "probability": {
              "false": 0.03387957196040836,
              "true": 0.9661204280395916
            }
          }
        }
      }
    }
  }
}

Meanwhile this is what we return from LiftWing:

{"predictions": {"prediction": true, "probability": {"false": 0.03387957196040836, "true": 0.9661204280395916}}}

I would personally vote to change a little our model.py servers to return the ORES response, so that migrating tools to Lift Wing will end up easier. If the output is the same then we'll only need to change the endpoints.

Event Timeline

Change 839516 had a related patch set uploaded (by AikoChou; author: AikoChou):

[machinelearning/liftwing/inference-services@main] editquality: align ORES prediction output with Lift Wing's one

https://gerrit.wikimedia.org/r/839516

Change 839516 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] editquality: align ORES prediction output with Lift Wing's one

https://gerrit.wikimedia.org/r/839516

Current status:

  • changed the model.py server of editquality to return output that aligned the ORES prediction output.
  • the new Docker image was rolled out and tested in staging.

Next steps:

  • apply the same change to articlequality, draftquality, and topic model servers.

Change 842358 had a related patch set uploaded (by AikoChou; author: AikoChou):

[machinelearning/liftwing/inference-services@main] draftquality: align ORES prediction output with Lift Wing's one

https://gerrit.wikimedia.org/r/842358

Change 842360 had a related patch set uploaded (by AikoChou; author: AikoChou):

[machinelearning/liftwing/inference-services@main] topic: align ORES prediction output with Lift Wing's one

https://gerrit.wikimedia.org/r/842360

Change 842362 had a related patch set uploaded (by AikoChou; author: AikoChou):

[machinelearning/liftwing/inference-services@main] articlequality: align ORES prediction output with Lift Wing's one

https://gerrit.wikimedia.org/r/842362

Change 842362 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] articlequality: align ORES prediction output with Lift Wing's one

https://gerrit.wikimedia.org/r/842362

Change 842360 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] topic: align ORES prediction output with Lift Wing's one

https://gerrit.wikimedia.org/r/842360

Change 842358 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] draftquality: align ORES prediction output with Lift Wing's one

https://gerrit.wikimedia.org/r/842358

The output change has been applied to all other revscoring model servers.

Going to mark this as RESOLVED. Feel free to reopen if there are any tasks I missed.