Page MenuHomePhabricator

Make wp10 rows be a squeezed to a weighted sum in ores_classification
Closed, ResolvedPublic

Description

If we deploy wp10 storage, it will take lots of space and also lots of work as each class will take one row per revision, SP team decided to keep a squeezed version of it in the database that is called weighted sum and we used it before too. See this mock: https://gist.github.com/halfak/b925a2d45a3903a3e10dc5d6cd7c01b1

Event Timeline

Ladsgroup created this task.May 9 2018, 6:37 PM

Change 434689 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/ORES@master] Make squeezing of score results possible so they take less rows

https://gerrit.wikimedia.org/r/434689

I like this, but I think we should use the term "aggregate" rather than "squeeze".

It seems that this metaphor works for damaging and goodfaith too. There, we want to aggregate two probabilities into one row.

E.g. a "class_selector(true)" aggregator would turn {"probability": {"true": 0.95, "false": 0.05}} into damaging 0.95

Similarly a "weighted_sum(weights)" aggregator would turn {"probability": {"Stub": 0.34, ..., "FA": 0.01}} into weighted_sum 1.67

It seems that this metaphor works for damaging and goodfaith too. There, we want to aggregate two probabilities into one row.

E.g. a "class_selector(true)" aggregator would turn {"probability": {"true": 0.95, "false": 0.05}} into damaging 0.95

Similarly a "weighted_sum(weights)" aggregator would turn {"probability": {"Stub": 0.34, ..., "FA": 0.01}} into weighted_sum 1.67

We already store one row for damaging classes because you can get the other one by subtracting from 1

Right. I guess what I was getting at is that storing one row is already a form of aggregation (selection).

Change 434689 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Make aggregation of score results possible so they take less rows

https://gerrit.wikimedia.org/r/434689

Change 437295 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/ORES@master] Make ScoreParser aggregate wp10 predictions

https://gerrit.wikimedia.org/r/437295

Change 437295 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Make ScoreParser aggregate wp10 predictions

https://gerrit.wikimedia.org/r/437295

Vvjjkkii renamed this task from Make wp10 rows be a squeezed to a weighted sum in ores_classification to 69caaaaaaa.Jul 1 2018, 1:10 AM
Vvjjkkii removed Ladsgroup as the assignee of this task.
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: Aklapper, gerritbot.
Vachovec1 renamed this task from 69caaaaaaa to Make wp10 rows be a squeezed to a weighted sum in ores_classification.Jul 1 2018, 3:44 PM
Vachovec1 raised the priority of this task from High to Needs Triage.
Halfak added a comment.Oct 9 2018, 7:12 PM

Looks like this is done.

Ladsgroup closed this task as Resolved.Nov 1 2018, 7:59 PM