Page MenuHomePhabricator

Search for every word in a query
Closed, ResolvedPublic

Description

Issue:
The Search function in the KaiOS app searches for article titles. In our usability studies, we saw that users use several words to form a query, which would result in a "no results page" error.

Request:
Allow for all words in a user's search query to show results.

  • Prioritize titles
  • If there is no title associated with the words a user has provided, show results based on article content
  • Let's ignore conjunctions and punctuation marks for now (I assume this is done already)

Note: changing Prefix to Fulltext api
Fulltext search api docs : https://www.mediawiki.org/wiki/API:Search
Prefix search api docs : https://www.mediawiki.org/wiki/API:Prefixsearch

Event Timeline

AMuigai created this task.Nov 25 2020, 2:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 25 2020, 2:33 PM

We're currently using prefixsearch with the following params

const params = {
    action: 'query',
    prop: ['description', 'pageimages', 'pageprops'].join('|'),
    piprop: 'thumbnail',
    pilimit: 15,
    ppprop: 'displaytitle',
    generator: 'prefixsearch',
    redirects: true,
    pithumbsize: 64,
    gpslimit: 15,
    gpsnamespace: 0,
    gpssearch: term.replace(/:/g, ' ')
  }

I think we stole it from the Android or iOS app. It works well for "zeroing in" on an article using part of the title but it's not a search like our users are expecting. We would like to cast wider and always have some results even if the relevance is getting low.

@EBernhardson do you have any recommendation about which search api/params would serve us better here? thanks!

Prefixsearch does exactly what it says, it searches for prefixes. If you want to use fulltext search, its something like:

{
  "action": "query",
  "list:" : "search",
  "srsearch": "my words to look for",
}

I suspect though that this is already known, and was decided to be not fit for purpose. Could you expand on this?

Yes, using Prefixsearch was on purpose but I didn't know srsearch was fulltext. We'll try it out in the app. I guess I could have read the docs.

hueitan claimed this task.Dec 1 2020, 10:23 AM
hueitan moved this task from Ready for Dev to Dev on the Inuka-Team (Kanban) board.
EBernhardson added a comment.EditedDec 1 2020, 4:25 PM

Yes, using Prefixsearch was on purpose but I didn't know srsearch was fulltext. We'll try it out in the app. I guess I could have read the docs.

Using prefix search will always treat the input as a prefix, the underlying data structures that allow the prefix search to efficiently look for fuzzy matches mean it can only ever start a search from the first character and proceed forward one character at a time (think of it like a DAG of utf8 codepoint transitions).

The srsearch parameter is only valid against list: search or generator: search, which triggers the fulltext search functionality.

I would generally suggest you implement something similar to what the website's do, for consistency of UX (although I'm certainly no UX expert). On the website both actions are used, depending on context. When the user is typing the content is sent to prefix search. The intent here is to complete the query prefix the user has provided. When submitting we first perform a near match search(srwhat: nearmatch provided to list: search). If the near match returns a page we take the user directly to that page and skip the search results page. If the near match fails we perform a full text search and show the results.

AMuigai triaged this task as Medium priority.Dec 1 2020, 6:02 PM
hueitan added a comment.EditedDec 7 2020, 6:02 AM

I would generally suggest you implement something similar to what the website's do, for consistency of UX (although I'm certainly no UX expert).

this is what we are doing now, but
it has a slight difference in the KaiOS search result (pilimit gpslimit set to 15)

Web SearchKaiOS Search (same as Mobile Web Search)

@AMuigai Could you tell us some of the search term example?

I would generally suggest you implement something similar to what the website's do, for consistency of UX (although I'm certainly no UX expert).

this is what we are doing now, but
it has a slight difference in the KaiOS search result (pilimit gpslimit set to 15)

Web SearchKaiOS Search (same as Mobile Web Search)

That's only the first stepof the mobile interface, where we attempt to expand the query prefix into a full query. The nearmatch and fulltext search is what happens when the search string is submitted as the fulltext search query, on mobile the user is sent here for the nearmatch and fulltext stage: https://m.mediawiki.org/w/index.php?search=cat

hueitan added a comment.EditedDec 9 2020, 1:03 PM

Thanks @EBernhardson

This is useful! I don't know there is the addition query after the submission in mobile web page until your comment here!

the request url : https://en.m.wikipedia.org/w/index.php?search=bread+recipes&title=Special%3ASearch&profile=default&fulltext=1&ns0=1

It has the result we are looking for, but it seems like it doesn't come from the public api, is this only available within mediawiki? (UPDATE: I have found the answer from the previous comment)

Prefixsearch does exactly what it says, it searches for prefixes. If you want to use fulltext search, its something like:

{
  "action": "query",
  "list:" : "search",
  "srsearch": "my words to look for",
}

My bad, I didn't reach this comment before, i think this is what we are looking for, let me try more on it.

https://en.m.wikipedia.org/w/api.php?action=query&list=search&srsearch=bread%20recipes&format=json

hueitan added a comment.EditedDec 10 2020, 10:51 AM

The differece between mobile web and kaios is no submit button (same to ios)

what I suggest :

  1. do the prefix search first, once receive no data from prefix then do the fulltext search
  2. do the fulltext search when prefix search result was given only few (exact number to be confirmed)

Note: currently there isn't search indicator, having another search api request require more time from the user side given the slow connection.


UPDATE: After a speak with Stephane, let's try with the fulltext without prefix for our initial test.

(PR https://github.com/wikimedia/wikipedia-kaios/pull/298)

hueitan moved this task from Dev to Code Review on the Inuka-Team (Kanban) board.Dec 16 2020, 12:46 PM
hueitan updated the task description. (Show Details)Dec 17 2020, 11:39 AM

@hueitan the PR is still a draft but this task is in "code review". What's the status?

SBisson moved this task from Code Review to QA on the Inuka-Team (Kanban) board.Mon, Jan 4, 2:58 PM
hueitan updated the task description. (Show Details)Tue, Jan 5, 9:55 AM
Jpita added a subscriber: Jpita.Mon, Jan 11, 10:16 PM

something doesn't look quite right here, look at the "page description" (line under the title of the page in the search page) comparison between kaios and mobile web

kaios before this taskkaios after this taskmobile web

What we are showing on the second line is not page description but a snippet showing where the search term was found in the article.

this is expected behaviour to show snippet in the description line.

AMuigai closed this task as Resolved.Tue, Jan 19, 9:03 AM