Page MenuHomePhabricator

UX check RC Filters in beta (revscoring 2.0/thresholds release)
Closed, ResolvedPublic

Description

The new thresholds selection system for the ORES extension is now deployed to Beta and seems to be working correctly from a technical point of view. It would be good to also ensure that it is working correctly from a UX point of view.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Halfak in betalabs ores_model, oresm_name for damaging was changed to 14. It seems to unable to display previously scored records. So, now RC page won't show any ORES scored for damaging for max number of 30 days.

And another question: there was a simple way to figure out the current OresDamagingLevels but I do not remember how...

@Halfak in betalabs ores_model, oresm_name for damaging was changed to 14. It seems to unable to display previously scored records. So, now RC page won't show any ORES scored for damaging for max number of 30 days.

We can manually change those in the database of beta cluster.

And another question: there was a simple way to figure out the current OresDamagingLevels but I do not remember how...

Go to deployment-tin.eqiad.wmflabs, mwscript eval.php --wiki=enwiki and then "echo $wgOresDamagingLevels;" This is what I do, if there is an easier way, please do tell me. It's already of work :D

Thx, @Halfak!

So, changed oresm_name is just the beta cluster thing, and in production it's handled differently? I mean, the only concern I have, it's that in production there would not be the same issue with "lost" ORES scores?

Just to be clear, @Ladsgroup != @Halfak. :) Thanks for getting to this, @Ladsgroup.

Sorry, should've paid attention :) and thx, @Ladsgroup. Anyway, my last question in my last comment was for @Halfak.

I think that @Ladsgroup or @awight might be better at answering your questions about MediaWiki-extensions-ORES

To clarify: @Etonkovidova's concern is that, because of the version change, ORES scores that predate the version change will not be found by the Extension:ORES/Recentchanges code. So if you filtered RC for "very likely bad", it would only find bad edits that happened after the version change, not before.

I see three options:

  1. Do nothing. Pre-version change edits will be treated as if they have no score
  2. Somehow include the old version of the model in the query. This will apply the new model's thresholds against the old model's scores, which may produce strange results.
  3. Re-score the latest N edits against the new model, similar to what we do when we first deploy ORES on a wiki. Each edit would then have two rows in the table, one with the old model score (ignored) and one with the new model score (used). I wonder if there's any caching that would get in the way if we tried to do this.

I think #3 might be the best approach.

Agreed. I believe that we have a maintenance script that is commonly run along with new deployments to score the last N edits with a new model. It seems that we should probably include running this as part of the process for deploying models with new versions.

Checked in betalabs - no issues found (there is very little data set overall).

Note:
Just general observations: checking ORES filters in wmf.6 (plwiki and enwiki) - quite few RC records fit simultaneously into several ranges of OresDamagingLevels ("maybebad", "likelybad", "verylikelybad"). It seems to be happening much more than before.
For example, the following https://en.wikipedia.org/w/index.php?title=List_of_noodles&curid=42443982&diff=809224555&oldid=809200544 will fall into all three ranges of damaging filters.

Also, there are more than usual records that fetched with "Very likely bad faith" and "Very likely good" filters.
For example, https://pl.wikipedia.org/w/index.php?title=Dyskusja_wikiprojektu:Polski_kanon_Wikipedii&curid=990017&diff=50853688&oldid=50853675

Although all of the above are not bugs, it'd be preferable to have such examples much less often.