Page MenuHomePhabricator

QA the phrase matching approach
Closed, ResolvedPublic

Description

One concern with the phrase matching approach T261398: Add a link engineering: Prototype the recommended link plugin in VisualEditor using phrase matching approach we've used for mapping link recommendations from the [link recommendation service]() to context items in VisualEditor is that wrapped entities may not get found in the VisualEditor document (see T267695: Add a link in VE: handle span-wrapped entities and annotations when finding link recommendation phrases for some examples – for example we might be searching for German cuisine but <i>German</i> cuisine is in VisualEditor).

We will be instrumenting how often phrase are not found in production via T278487: Instrumentation: Number of phrases not found in document, but before we go live with this feature it would be useful to do some QA with our target wikis.

Proposed test:

  • Visit 10 link-recommendation tasks on each target wikis beta instance
  • take note of how many instances there are where the number of expected phrases to be found doesn't match the actual of RecommendedLinkContextItems shown in the VisualEditor document.
    • for example, if the phrase café is in the article text, and the list of recommendations from the service includes a recommendation with the link text of café, but the the word café is not highlighted in blue with the robot icon in the VisualEditor document, then that is a problem.

We might want to wait for T278487: Instrumentation: Number of phrases not found in document, because we could for example output in the console when expected/actual doesn't match, to simplify the job for QA.

Event Timeline

@kostajh -- is this something the ambassadors could have picked up on when they did T278864: Add a link: evaluate link recommendation (Mar 30 2021), in which they were just looking at API outputs? Perhaps they have some memory of whether they saw this frequently or not.

Otherwise, if we go the route you suggested, I have these questions:

  • Do we have link suggestions available in all four pilot betas? ar, cs, bn, vi?
  • Are the articles with those link suggestions a good sample, such that we could extrapolate to outside of beta?
  • Which numbers should the testers compare? Like, how do they know how many matches the API wanted to find, but couldn't?

@kostajh -- is this something the ambassadors could have picked up on when they did T278864: Add a link: evaluate link recommendation (Mar 30 2021), in which they were just looking at API outputs? Perhaps they have some memory of whether they saw this frequently or not.

No, because this is evaluating whether our code is successfully finding the phrase or not. Thinking about this some more, we have T278487: Instrumentation: Number of phrases not found in document as a planned task that will do this calculation for us automatically. In that task we could implement logging to logstash when there is a problem (in addition to event logging events) so that we'll know when there are problems. If we go that route, the task for QA testing would be to just ensure that X number of link recommendation tasks (10? 20?) are done in each beta wiki so that we have enough data to work with to see if there is a problem or not. That would be substantially faster than asking the ambassadors to manually compare the API output with what they see on the page (more detail below).

Otherwise, if we go the route you suggested, I have these questions:

  • Do we have link suggestions available in all four pilot betas? ar, cs, bn, vi?

We do, yes.

  • Are the articles with those link suggestions a good sample, such that we could extrapolate to outside of beta?

I think so, yes.

  • Which numbers should the testers compare? Like, how do they know how many matches the API wanted to find, but couldn't?

In the browser console, you can paste this code mw.config.get( 'wgGESuggestedEditData' ) and press "enter", and it will show you the raw recommendations that we then attempt to place in the VisualEditor document.

@kostajh -- I think I understand, and it sounds to me like someone doesn't actually need to know the language in order to figure this out. Therefore, I think this is something @Etonkovidova could do: go to articles in each beta wiki, run the browser console code, compare to the count of highlighted links in the UI, record in spreadsheet. Is that right?

@kostajh -- I think I understand, and it sounds to me like someone doesn't actually need to know the language in order to figure this out. Therefore, I think this is something @Etonkovidova could do: go to articles in each beta wiki, run the browser console code, compare to the count of highlighted links in the UI, record in spreadsheet. Is that right?

Yeah, sounds right. But I would also add that in addition to comparing the count, it would be helpful to verify that the "link_text" value from the console snippet is exactly what is highlighted in the document, e.g. that sovětský appears highlighted as an annotation in the document and not sovětsk

grafik.png (163×891 px, 53 KB)

kostajh changed the task status from Open to Stalled.Apr 26 2021, 7:52 AM
kostajh assigned this task to Etonkovidova.
kostajh edited projects, added Growth-Team (Current Sprint); removed Growth-Team.

Moving into current sprint and assigning to Elena, but also marking as stalled until the patch for T267706: Add a link in VE: Use a permanent context instead of VE's built-in context is merged. In theory, the work in T267706 shouldn't affect this QA task, but better to be certain and wait to QA this after the permanent context patch is in beta.

kostajh changed the task status from Stalled to Open.Apr 27 2021, 8:29 PM

The patches in T267706: Add a link in VE: Use a permanent context instead of VE's built-in context have been merged and should be on betalabs shortly, so this task is ready to go.

Checked arwiki, cswiki, viwiki, and bnwiki. The testing was done per the task description.

To summarize
(1) No discrepancies between the number of expected phrases to be found and the actual ones.
(2) arwiki and bnwiki displayed "error: "Link suggestion not found" only once per ten articles.
e.g. for https://ar.wikipedia.org/w/index.php?title=كنيسة_سيستينا&veaction=edit&action=edit

Link suggestion not found for "كنيسة سيستينا" load.php:234:876
    require https://ar.wikipedia.org/w/load.php?
    runScript https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:12
    cssHandle https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    flushCssBuffer https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:4
    addEmbeddedCSS https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:5
    execute https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    doPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:6
    (Async: requestIdleCallback handler)
    requestPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    setAndPropagate https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    markModuleReady https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:12
    runScript https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    cssHandle https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    flushCssBuffer https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:4
    addEmbeddedCSS https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:5
    execute https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    doPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:6
    (Async: requestIdleCallback handler)
    requestPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    setAndPropagate https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    markModuleReady https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:12
    runScript https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    cssHandle https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    flushCssBuffer https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:4
    addEmbeddedCSS https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:5
    execute https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    doPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:6
    (Async: requestIdleCallback handler)
    requestPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    setAndPropagate https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    markModuleReady https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:12
    runScript https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    cssHandle https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    flushCssBuffer https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:4
    addEmbeddedCSS https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:5
    execute https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    doPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:6
    (Async: requestIdleCallback handler)
    requestPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    setAndPropagate https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    markModuleReady https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:12
    runScript https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    cssHandle https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    flushCssBuffer https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:4
    addEmbeddedCSS https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:5
    execute https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:13
    doPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:6
    (Async: requestIdleCallback handler)
    requestPropagation https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    setAndPropagate https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:7
    implement https://ar.wikipedia.org/w/load.php?lang=ar&modules=startup&only=scripts&raw=1&skin=vector:19
    <anonymous> jQuery

cswiki displayed the above error twice.