Page MenuHomePhabricator

Document JADE judgment structure
Closed, ResolvedPublic

Description

Wiki page with a description of the data contained in a JADE judgment. Note that some documentation already exists in the JADE JSON schemas.

See: https://www.mediawiki.org/wiki/JADE/Schema

Event Timeline

Halfak created this task.Oct 30 2017, 3:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 30 2017, 3:07 PM
Halfak triaged this task as Medium priority.Nov 1 2017, 4:56 PM
Halfak added a project: good first task.
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptNov 1 2017, 4:56 PM
Dheerajmalisetty raised the priority of this task from Medium to High.Jan 12 2018, 2:56 PM
Dheerajmalisetty lowered the priority of this task from High to Medium.
Halfak added a subscriber: awight.Feb 14 2018, 4:23 PM

@awight, do you think we still need to do this?

I think we should still document what's happening in a JADE judgment, lemme rewrite the task.

awight renamed this task from Mock JADE discussion page to Document JADE judgment structure.Feb 14 2018, 4:25 PM
awight updated the task description. (Show Details)

It's awkward that judgments and endorsements don't both have the author metadata, it seems difficult to extract the same information from the edit log.

Halfak assigned this task to awight.Jun 18 2018, 1:49 PM

Both should have guid. We should be able to extract author information from the API or elsewhere. It's sort of like storing only the Qid in a wikidata statement because the label could change.

Both should have guid. We should be able to extract author information from the API or elsewhere. It's sort of like storing only the Qid in a wikidata statement because the label could change.

This sounds good to me, let's normalize away the user names.

I'd like to add something about our endorsements conversation today but don't have any tangible proposals yet. The cycle of logic I've been rehashing is, * judgments without endorsements will deliver immediate value, so an endorsement feature can be delivered separately. The schema change is backwards-compatible if we do decide to go with the endorsements as envisioned now. User testing will confirm our workflows soon enough. * On the other hand, there's not much extra complexity, it enables a likely workflow and pro-social behavior, * If we leave communication and consensus entirely to talk pages, these still have enough structure that we could mine them for much richer semantics than even the endorsement schema, for example direct replies and other interactions, support-oppose-neutral distinctions of any type, etc.

How about this proposal:

  • Add optional comment, guid, and origin fields to judgments. judgment.comment is intended to hold a justification of the judgment, and the edit comment is where to record notes about the change itself, such as your motivation for adding or editing the judgment. For example, "Provide a counterexample to https://ores.wikimedia.org/v3/scores/enwiki/123456?models=drafttopic", or "Fix epic typo in the justification".
  • Drop endorsements for now, and suggest that these be adding to the Jade_talk page using *{{s}} or any other discussion style.
  • However, multiple judgments may have the same schema data, which already gives us a structured storage for the original endorsement use cases, if they prove natural and common in Talk pages. It also reduces the conceptual overhead of explaining the difference between a judgment and an endorsement.

If we keep endorsements, I think the origin might make more sense in the endorsement since you could get endorsements from different origins and that would be meaningful. What do you think?

If we drop endorsements, then what does it mean to have "origin" in the Judgement if many people could agree on that Judgement?

I'm still interested in dropping endorsements and leaving them in the talk page. I like the natural split. One thing that is really common in straw poles/!votes is to *respond* to someone's endorsement with questions and comments. Sometimes this leads to someone changing their endorsement. If we can handle the "origin" case in some reasonable way, then I'm down for moving them to Talk.

Okay, let's drop endorsements and move to talk.

I don't quite catch what you're saying about "origin", I am proposing that the judgment is accompanied by original author info, which can be used in any of these ways:

Bare judgments:

...
    {
        "data": true
    }

Judgment with original author attribution:

...
    {
        "data": true,
        "guid": 123456,
        "comment": "foo",
        "origin": "Huggle 3.4.3"
    }

Redundant judgments with authors making complementary points:

...
    {
        "data": true,
        "origin": "Huggle 3.4.3",
        "guid": 123456,
        "comment": "foo"
    },
    {
        "data": true,
        "guid": 654321,
        "comment": "bar"
    }

Arg. This just looks like endorsements in a messy structure to me. :|

If we're dropping endorsements for the first iteration, let's also not try to simulate it using the multiple judgment pattern in my last example, then, and just ignore that case. I was just showing that our schema without endorsements still makes it possible to build that feature, for experimentation or whatever.

More importantly, what do you think about origin, guid, and comment being properties of a judgments? Since I still don't understand your concern about "origin" in T179301#4306270, I probably misunderstand the purpose of the field. I was thinking of it as a user-agent for anything that integrates with JADE.

Re. origin. I'm imagining the following scenario:

  1. A huggle user marks an edit as damaging and bad-faith
  2. A judgement is created for the edit and the origin is set to "Huggle <version>"
  3. Another user is reviewing revert actions using Special:Diff and decided that the same edit is damaging, but good-faith
  4. This user changes the judgement (confirming the damaging, but changing the good-faith value)

The origin for "goodfaith" would likely change from "Huggle <version>" to "Special:Diff". But what would the origin of "damaging" say? I assume it would still say "Huggle <version>" even though the judgement was confirmed by someone using Special:Diff.

As a data consumer, I want to be skeptical of the labels that come from quality control tools that use ORES due to the bias. However, if a judgement is confirmed (or changed) through WikiLabels or maybe the default diff interface, then I'll trust that judgement more.

Maybe the pseudo-user-agent should go into Change Tags, or be prepended to the edit comment.

Just noting that a data consumer is probably reading the eventbus feed for the JADE namespace, so they'll have to make some API requests to get the judgment content. This hypothetical bot will have no problem extracting the judgment author's identity, change tags, etc. The only argument I can see for putting anything other than judgment.data into the page content is if users might want to revise it later. origin and guid never have to be revised, and in fact cause problems with our data model as you point out, since they aren't naturally 1:1 with judgments.

Mini-consensus in IRC was to have judgments with only "data" and "notes" fields, and drop endorsements for now. We'll encourage collaborative revision of "notes".

Change 442990 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/JADE@master] Fix schema: syntax of boolean data

https://gerrit.wikimedia.org/r/442990

Change 442884 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/JADE@master] Drop entity from the schema

https://gerrit.wikimedia.org/r/442884

Change 442990 merged by jenkins-bot:
[mediawiki/extensions/JADE@master] Fix schema: syntax of boolean data

https://gerrit.wikimedia.org/r/442990

Change 442884 merged by jenkins-bot:
[mediawiki/extensions/JADE@master] Drop entity from the schema

https://gerrit.wikimedia.org/r/442884

awight closed this task as Resolved.Aug 7 2018, 6:45 PM
awight removed a project: Patch-For-Review.