For example, consider the following query:
All of the search results that are not redirects correctly have an "index" field that specifies the result's position in the list that is shown to the user. However, the results that are redirects do not have an "index," so a consumer of the API no longer knows in which position to display these results.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dbrant | T104746 Search results should appear in a sensible order | |||
| Resolved | EBernhardson | T92796 Prefix search API doesn't return "index" field when "redirects" is enabled. |
This is intentional, because doing anything else raises a lot of questions:
- What index should be returned for the redirect target?
It would be the index that the un-followed redirect result would have had.
Intuitively I would expect to see something like the following:
If there is any duplication in the results due to the redirect target, we already combine duplicates into a single result (from doing prefixsearch and full-text search simultaneously), so it won't be an issue.
And what about all the rest of the questions?
If the "redirects" parameter is enabled, the returned prefixsearch results would remain the same
Sorry, that's not at all how the 'redirects' parameter works.
If generator=prefixsearch returns 5 pages that all redirect to the same target, the API result is going to have one entry for the target page, not 5. If you want each individual redirect returned, then don't specify 'redirects', but then it's not going to give you what you probably want for prop=pageterms.
What currently influences the presence of redirects and the order in which they appear? The results seem pretty consistent but not fully deterministic. For example, the following query sometimes returns "Obama" at the top of redirects list, but other times returns "Obama crush" as the first redirect:
And when "Obama crush" isn't the first redirect on the list, it typically isn't on the list at all...
The underlying search engine determines which results are returned from prefixsearch. It does not necessarily give the same set of items every time for the same query, and in particular items might move across the 'limit' boundary.
Items in the 'redirects' array in the result are not intentionally ordered. They're in whatever order the database chooses for the unordered bulk query of the redirect targets for all pending redirects in the result set, which is not required to be deterministic or consistent. For MySQL/MariaDB it's likely to use the primary key on the redirect table and thus will happen to return the redirect rows in page_id order.
So just to clarify, is there no way to preserve the index parameter from prefixsearch when resolving redirects? (i.e. include the original index with the redirected result) (In the case of multiple prefixsearch results pointing to the same redirect, the index can be that of the first prefixsearch result in the set)
Or, a broader question: How can we ascertain the intended order of our search results when redirects=true?
Change 230521 had a related patch set uploaded (by EBernhardson):
Keep 'index' in the result set when resolving redirects
What if we add redirect handling 'modes'? With no value or an unknown value in the redirects parameter you get the default handling (perhaps with a warning for unknown values), specifying redirects=preserve-index (or some such name) would do as is requested here? Or if overloading redirect is undesired a new query param can be defined.
I think that the behaviour requested here should be the default behaviour; there is an expectation if you request a list of search results for a given query that the results be ordered. I'm fine with adding an API parameter for the old behaviour.
@Deskana the problem with making anything the default parameter is that this redirects parameter is not specific to the search query, it is an API wide parameter that is already widely used.
@Anomie another option, what if we merged the generator data into the entry in 'redirects', then the client still gets the index and there is no 'random' decision happening in the api by chance of ordering.
I don't want us to build up to the state where when we're making recommendations to third parties about how to use our search API that we have to say "...and stick all these parameters on the end to make it behave like an actual sensible search API".
That said, Discovery's priority is higher for users of Wikimedia sites higher than for third party API users, so if we must add an API parameter for this to get a fix working fast, then we should do that.
Change 230585 had a related patch set uploaded (by EBernhardson):
Include generated metadata for redirects
Changing priority to high to reflect the fact that Discovery-ARCHIVED is at work on this.
looks like this might get pushed back till next week, the primary api maintainer is on vacation.
Change 230521 abandoned by DCausse:
Keep 'index' in the result set when resolving redirects
Reason:
This solution won't work, sorry for the noise.
David's patch was abandoned, but there is a patch from @EBernhardson that needs review once @Anomie is back from vacation: https://gerrit.wikimedia.org/r/230585
This is now completed in terms of the API returning the appropriate information (in next weeks train deploy). The relevant consumers need to now be updated to use this information.
This needs documentation for how API consumers can update their clients. This is tracked in T110231.
Given the discussion taking place on T110231 where the end users of this change are unsatisfied, I am reopening this task for further review.
The stuff they're wanting there isn't going to happen, so I'd say just close this as either Resolved or Declined. I don't care which.
I'm sorry. What? Yes it will. If we need to build a new API cluster to fix architectural mistakes, so be it, but "I don't like it" doesn't trump reality.
"I want it" doesn't trump the fact that it's not logically possible without special-case hacks all over the place either.
For the sake of the people in this discussion that aren't completely familiar with how the Action API works for a query like the Obama example, I'm going to try and give a basic outline. I just traced through all of this today for the first time, so if I get a few bits wrong here and there hopefully @Anomie and @EBernhardson can jump in to correct them.
This is the api query I'm going to walk through:
https://en.wikipedia.org/w/api.php?action=query&prop=pageterms&generator=prefixsearch&gpssearch=obama&gpslimit=20&redirects&continue
The query string here says:
The use of generator=prefixsearch throws in an implicit requirement as well:
When api.php reads this set of instructions it does roughly this behind the scenes:
So at the point that the results are output, we have a collection of pages which will have an index if they were mentioned by Title in the ApiQueryPrefixSearch results and (possibly) a collection of from => to redirect pairs that were resolved by ApiQuery. The redirect pairs will also have index data if the from Title appeared in the ApiQueryPrefixSearch results.
From the point of view of the generic ApiQuery module which can be driven by a number of different generator data sets, this is all a generic process. It just asks the generator for a list of titles, page ids or revision ids and uses that to find pages.
The hook points that are used to decorate the results with index data from the ApiQueryPrefixSearch is generic as well. Any generator can store any information based on a Title that it wants to have merged into the final page information and it will be merged in the same way. The lack of special handling for particular combinations of generators and properties makes the backing system relatively easy to maintain and extend.
The discussion after T110231#1575192 is asking for a perfectly reasonable sounding fix for the complexity of merging data from the redirects collection into the pages collection when generator=prefixsearch and redirects are used together from the perspective of an API consumer who is not using a higher level abstraction to manage the complexity of talking to the Action API. These users want a simple way to accomplish their end goal of a list of pages ordered by search relevance. This is the sort of nicety that an Action API wrapper (eg Pywikibot) generally provides.
As the current keeper of sanity and fixer of bugs in the whole Action API system, @Anomie is coming to this discussion with a perspective of wanting implemented solutions to be generic and to work properly (or at least consistently) no matter what combination of generators and properties are requested. From his point of view this isn't a problem getting ordered search results in a consuming application; it is instead a problem of deciding if there is a safe and sane way to fulfill the feature request at the general purpose API level that will be beneficial to multiple use cases or at the very least not detrimental to multiple use cases and painful to maintain.
If we can redirect the discussion back to attempting to decide if there are indeed universal answers to the questions @Anomie posed in T92796#1121265 or some other generic change to the Action API that would allow the desired result we have some chance of a productive outcome. There will be no productive result if the discussion degenerates into attacks on the basic infrastructure or @Anomie's duty to protect our ability to maintain and extend the system.
As a general description it's accurate. There are some minor detail errors, e.g.
and (possibly) a collection of from => to redirect pairs that were resolved by ApiQuery.
it's actually ApiPageSet that resolves the redirects. I could go through and flag all of these, but I'm guessing that would be excessive detail that isn't really relevant to the discussion at hand.
As the current keeper of sanity and fixer of bugs in the whole Action API system, @Anomie is coming to this discussion with a perspective of wanting implemented solutions to be generic and to work properly (or at least consistently) no matter what combination of generators and properties are requested. From his point of view this isn't a problem getting ordered search results in a consuming application; it is instead a problem of deciding if there is a safe and sane way to fulfill the feature request at the general purpose API level that will be beneficial to multiple use cases or at the very least not detrimental to multiple use cases and painful to maintain.
I endorse this summary of my viewpoint.
So how can we move this forward? One idea could be setting some sort of merge policy on the ApiPageSet? ApiQueryPrefixSearch receives the ApiPageSet as an argument. it could do something like:
$pageSet->setRedirectMergePolicy( function ( $current, $new ) {
if ( !isset( $current['index'] ) || $new['index'] < $current['index'] ) {
return $new;
}
return $current;
} );The default merge policy would always return $current which retains current behavior. Generators that want to have data merged can set their own policy as above. I havn't written any actual code so there might be some problems with this idea...
I spoke with @Anomie, and while he would prefer a clientside solution, and has concerns about api complexity, he does not object to @EBernhardson's merge policy proposal.
Nice. @EBernhardson, can you or one of the other devs take a stab at implementing this? @TJones was looking for work yesterday, perhaps he could pick it up.
Change 243908 had a related patch set uploaded (by DCausse):
Allow generators to keep track of resolved redirects
Change 243908 abandoned by DCausse:
Allow generators to keep track of resolved redirects
Reason:
Won't work (thanks for your time).
Change 244348 had a related patch set uploaded (by EBernhardson):
Implement ApiPageSet::setMergePolicy()