Page MenuHomePhabricator

Expand wiki search limit to be larger than 1000
Open, LowPublic

Description

Currently, AWB's listmaker will only return 1000 results from a wiki search. This can makes assembling larger lists require manual search result processing outside of AWB, since there is no way to fetch the next 1000 results. Performing fixes on the first batch of 1000 and then doing a new search doesn't always work because (a) the search results seem to lag behind after edits are made and (b) some fixes, especially those that modify non-alphanumeric characters, cannot be distinguished from the original text by the search engine.

This presumably dates back to when the API only returned 50/500 results for user/bots. However, now that the API limits have increased to 500/5000 results, it might make sense to expand AWB's limit to 10,000 results (coincidentally, for Wikis using CirrusSearch, 10,000 total results is the most that will be returned using the "continue" function).

This could either be done in AWB core, or in the NoLImits plugin.

Event Timeline

I note from https://en.wikipedia.org/w/api.php?action=help&modules=query%2Bsearch

srlimit
How many total pages to return.

No more than 500 (5,000 for bots) allowed.

So for users without apihighlimits, it would require 20 api queries

@Reedy Is 20 queries a problem? Wasn't that the status quo when the limit was 50/500?

If it is a concern, this could be rolled into the nolimits plugin which is restricted to users with apihighlimits anyway.

@Reedy Is 20 queries a problem? Wasn't that the status quo when the limit was 50/500?

Remember, on WMF wikis search isn't an SQL/DB query, it's going out to ElasticSearch. Would need to check in with Discovery-ARCHIVED and see if they're happy for us to push a few more queries their way

Aklapper triaged this task as Low priority.Feb 10 2023, 12:05 PM