Page MenuHomePhabricator
Create Task
Maniphest T416530

[SPIKE] Investigate technical feasibility of surfacing Reference Score suggestions within Suggestion Mode
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

This task involves the work of investigating what technical work would be involved in surfacing the reference survival signal (T276857) within the VisualEditor's Suggestion Mode.

Learning objectives

  • 1. What technical work would need to be done to enable the experience described in Requirements below?
  • 2. What (if any) technical work would the Editing Team need to depend on other teams to complete?
  • 3. What level of effort/complexity would you assign to the technical work the Editing Team would need to take on to enable the experience described in Requirements below?

Requirements

For the purposes of this investigation, surfacing the reference survival signal (T276857) within Suggestion Mode would result in the following user experience:

  1. When people open the visual editor on a wiki where they've enabled Suggestion Mode and meet the experiential requirements defined within the suggestion's config:
    1. Show a Suggestion "card" when the model returns a score that is below the threshold set within the suggestion's config
    2. E.g. presented in the side-rail (desktop) and gutter (mobile), using a card design that is consistent with other suggestions, with the content the suggestion is relevant to highlighted (in this case, the reference in question), etc.

Acceptance criteria

  • Answers to all "Learning objectives" are documented in this ticket's task description

References

Related Objects
Search...

Event Timeline

ppelberg created this task.Feb 4 2026, 10:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2026, 10:08 PM
ppelberg edited projects, added: Editing-team (Editing-2Mar-17Mar-2026); removed: Editing-team.Feb 4 2026, 10:08 PM
ppelberg added a parent task: T276857: Surface Reference survival signal within VE.
ppelberg updated the task description. (Show Details)Mar 2 2026, 5:52 PM
ppelberg added subscribers: Lina_Farid_WMDE, Pablo, diego, Sucheta-Salgaonkar-WMF.
ppelberg edited projects, added: Editing-team (Editing-18Mar-27Mar-2026); removed: Editing-team (Editing-2Mar-17Mar-2026).Mar 18 2026, 4:25 PM
ppelberg moved this task from Decisions to be made to Ready to Be Worked On on the Editing-team (Editing-18Mar-27Mar-2026) board.Mar 18 2026, 5:57 PM
VPuffetMichel set the point value for this task to 2.Mar 18 2026, 5:57 PM
ppelberg moved this task from Backlog to New Suggestions on the VisualEditor Suggestion Mode board.Mar 27 2026, 7:58 PM
ppelberg edited projects, added: Editing-team (Editing-Q4-30Mar-10Apr-2026); removed: Editing-team (Editing-18Mar-27Mar-2026).Mar 27 2026, 10:38 PM
ppelberg moved this task from Decision to be made to Ready to Be Worked On on the Editing-team (Editing-Q4-30Mar-10Apr-2026) board.Mar 27 2026, 10:50 PM
ppelberg triaged this task as High priority.Mar 27 2026, 10:57 PM
ppelberg lowered the priority of this task from High to Medium.Mar 31 2026, 9:33 PM
DLynch claimed this task.Apr 12 2026, 11:04 PM
DLynch moved this task from Ready to Be Worked On to Ready for Sign Off on the Editing-team (Editing-Q4-30Mar-10Apr-2026) board.
DLynch subscribed.

It took me some actual digging around to find this because the model card isn't one of the helpful ones that links to the service, so it's actually documented at https://api.wikimedia.org/wiki/Lift_Wing_API/Reference/Get_reference_risk_prediction (though api. is getting shut down, so presumably that link has a limited lifespan).

I tested this out via curl https://api.wikimedia.org/service/lw/inference/v1/models/reference-risk:predict -X POST -d '{"rev_id": 1347272125, "lang": "en", "extended_output": "True"}' | jq . which is asking for the individual reference details of this specific revision of enwiki's Castle.

The output I received was:

{
  "model_name": "reference-risk",
  "model_version": "2024-11",
  "wiki_db": "enwiki",
  "revision_id": 1347272125,
  "reference_count": 16,
  "survival_ratio": {
    "min": 0.6883097910027386,
    "mean": 0.874415039084355,
    "median": 0.8883447823800822
  },
  "reference_risk_score": 0.0,
  "references": [
    {
      "url": "https://books.google.com/books?id=8A1_Z1DTgYYC&pg=PA6",
      "domain_name": "books.google.com",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8883447823800822,
        "page_count": 583492,
        "editors_count": 63816
      }
    },
    {
      "url": "http://melitensiawth.com/incoming/Index/L-Imnara/L-Imnara.%2007(2002)1=26/05s.pdf",
      "domain_name": "melitensiawth.com",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.9272520761075003,
        "page_count": 294,
        "editors_count": 26
      }
    },
    {
      "url": "http://dinlarthelwa.org/wp-content/uploads/2012/05/41_Final.pdf",
      "domain_name": "dinlarthelwa.org",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.909028907584063,
        "page_count": 77,
        "editors_count": 21
      }
    },
    {
      "url": "https://books.google.com/books?id=m-TqPC6cRNYC&q=Postern&pg=PA17",
      "domain_name": "books.google.com",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8883447823800822,
        "page_count": 583492,
        "editors_count": 63816
      }
    },
    {
      "url": "https://books.google.com/books?id=GnBPAAAAMAAJ",
      "domain_name": "books.google.com",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8883447823800822,
        "page_count": 583492,
        "editors_count": 63816
      }
    },
    {
      "url": "https://whc.unesco.org/en/tentativelists/5282/",
      "domain_name": "whc.unesco.org",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8817681872709213,
        "page_count": 6669,
        "editors_count": 3249
      }
    },
    {
      "url": "https://brill.com/view/book/edcoll/9789004414365/BP000002.xml",
      "domain_name": "brill.com",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8332474308678378,
        "page_count": 3724,
        "editors_count": 2209
      }
    },
    {
      "url": "https://books.google.com/books?id=vkV8bcgLbiAC&q=the+celtic+world",
      "domain_name": "books.google.com",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8883447823800822,
        "page_count": 583492,
        "editors_count": 63816
      }
    },
    {
      "url": "https://whc.unesco.org/en/list/1229",
      "domain_name": "whc.unesco.org",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8817681872709213,
        "page_count": 6669,
        "editors_count": 3249
      }
    },
    {
      "url": "https://www.academia.edu/23266199",
      "domain_name": "academia.edu",
      "domain_metadata": {
        "ps_label_local": "No consensus",
        "ps_label_enwiki": "No consensus",
        "survival_ratio": 0.8121416554518694,
        "page_count": 25453,
        "editors_count": 11721
      }
    },
    {
      "url": "http://www.mnh.inah.gob.mx/historia/hist_historicos.html",
      "domain_name": "mnh.inah.gob.mx",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.9663088679363844,
        "page_count": 6,
        "editors_count": 5
      }
    },
    {
      "url": "https://books.google.com/books?id=jnVEsUqTtIgC&pg=PA53",
      "domain_name": "books.google.com",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8883447823800822,
        "page_count": 583492,
        "editors_count": 63816
      }
    },
    {
      "url": "https://www.theguardian.com/business/2011/may/30/playmobil-malta-theme-park",
      "domain_name": "theguardian.com",
      "domain_metadata": {
        "ps_label_local": "No consensus",
        "ps_label_enwiki": "No consensus",
        "survival_ratio": 0.9094254173213723,
        "page_count": 185332,
        "editors_count": 50522
      }
    },
    {
      "url": "https://www.kansallismuseo.fi/fi/haemeenlinna/historia",
      "domain_name": "kansallismuseo.fi",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.85789800536474,
        "page_count": 16,
        "editors_count": 12
      }
    },
    {
      "url": "https://www.youtube.com/watch?v=gia1B97H61U",
      "domain_name": "youtube.com",
      "domain_metadata": {
        "ps_label_local": "Generally unreliable",
        "ps_label_enwiki": "Generally unreliable",
        "survival_ratio": 0.6883097910027386,
        "page_count": 231207,
        "editors_count": 129224
      }
    },
    {
      "url": "https://whc.unesco.org/en/list/847",
      "domain_name": "whc.unesco.org",
      "domain_metadata": {
        "ps_label_local": null,
        "ps_label_enwiki": null,
        "survival_ratio": 0.8817681872709213,
        "page_count": 6669,
        "editors_count": 3249
      }
    }
  ]
}

The main inconvenience in using this will be matching up the references. It's a bit of a pain going through all the references and interrogating them to work out whether they contain a specific URL. Doable, though.

The model documentation suggests it should not be used for "Evaluating a domain that has no previous usage history in the target language Wikipedia", but I'm not sure it's possible to tell the difference between one that has never been used before and one that has been thoroughly removed from the wiki.

Like other models like add-a-link/image, this could never be used as a check. It relies on precomputing information about a given revision, and cannot be asked about unpublished content.

There's some open UX questions about how to do this because references can be reused. Do we show a suggestion on every usage and have them removed/replaced separately? Do we offer to replace every usage? (How does this interact with subreferences?)

Overall, I don't think an initial implementation is too much effort. Would take a day or two.

ppelberg edited projects, added: Editing-team (Editing-Q4-13Apr-24Apr-2026); removed: Editing-team (Editing-Q4-30Mar-10Apr-2026).Apr 13 2026, 4:33 AM

(Moving to upcoming sprint b/c there will not be space for me to process the thorough findings @DLynch helpfully documented above before it begins)

ppelberg moved this task from Decision to be made to Ready for Sign off on the Editing-team (Editing-Q4-13Apr-24Apr-2026) board.Apr 13 2026, 4:34 AM
ppelberg edited projects, added: Editing-team (Editing-Q4-27Apr-8May-2026); removed: Editing-team (Editing-Q4-13Apr-24Apr-2026).Apr 29 2026, 9:03 PM
ppelberg moved this task from Decision to be made to Ready for Sign off on the Editing-team (Editing-Q4-27Apr-8May-2026) board.Apr 30 2026, 5:20 PM
ppelberg closed this task as Resolved.May 13 2026, 12:21 AM

The main inconvenience in using this will be matching up the references. It's a bit of a pain going through all the references and interrogating them to work out whether they contain a specific URL. Doable, though.

T425808, which @DLynch helpfully filed, will (I think) help here.

The model documentation suggests it should not be used for "Evaluating a domain that has no previous usage history in the target language Wikipedia", but I'm not sure it's possible to tell the difference between one that has never been used before and one that has been thoroughly removed from the wiki.

Documented as Open question #1 in T418554.

There's some open UX questions about how to do this because references can be reused. Do we show a suggestion on every usage and have them removed/replaced separately? Do we offer to replace every usage? (How does this interact with subreferences?)

Documented as Open question #2 in T418554.

Johannnes89 subscribed.Jun 15 2026, 7:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits