SearchEngine::setLimitOffset() has an offset parameter, but translating an offset to a search_after value seems hard if at all possible. So I think the nicer thing to do would be to issue multiple queries internally within ISearchResultSet's iterator logic (so this is actually a CirrusSearch issue, not a core one).

Exposing this feature might involve a dedicated API, I'm not sure that the existing SearchEngine and ISearchResultSet is well suited for this:

• might be easier to just assume that we are going to sort on page_id, so that db engines might have a chance to be able to implement this.
• the client must be able to resume this operation so that it can be processed via multiple jobs (to avoid long running maint scripts)

All this is already possible using internal CirrusSearch APIs, have this solution be considered (see for instance https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ImageSuggestions/+/refs/heads/master/includes/Notifier.php)?

A DB engine should not have a problem paging on any set if criteria it can efficiently sort on, I think? It's slightly more effort as you have to manually write the query for each sort while CirrusSearch just gives you the continuation data to pass into the next request, but still not hard. I'd probably add a SearchEngine::setContinue() and an IContinuableSearchResultSet::getContinue(), and the caller could just check whether a IContinuableSearchResultSet was returned, and over time all implementations would migrate to returning that (which is unfortunately the least painful way of adding a new method to an interface).

In T347603#9216493, @dcausse wrote:

All this is already possible using internal CirrusSearch APIs, have this solution be considered (see for instance https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ImageSuggestions/+/refs/heads/master/includes/Notifier.php)?

I think that would work for T345713 specifically. But other aspects of structured tasks where this limitation can also be relevant rely on community-configurable search strings to filter the list of applicable tasks; so a CirrusSearch query string would have to be transformed to an Elastica query and then search_after added to that. That seems quite involved.

But mainly I think this would be a valuable addition in general that would prove useful in the future. Eg. we'd probably want to convert recent changes to CirrusSearch some day because currently it's a horror show of query planner edge cases that require lots of manual debugging and tuning, often differently for different DB engines or wiki profiles. But recent changes have more than 10K entries and some clients do want to go through the whole thing.

This will require a significant amount of design and coordination. As such, it should go through the Foundational Technology Requests prioritization.

If it's a lot of work, I don't think we have a sufficiently compelling use case. It seemed easily doable to me, but I don't know the codebase well.