Page MenuHomePhabricator

Search now caps total results count at 10k because of elasticsearch 7 upgrade
Closed, ResolvedPublic

Assigned To
Authored By
Legoktm
Sep 8 2022, 10:40 PM
Referenced Files
None
Tokens
"Like" token, awarded by Izno."Like" token, awarded by Multichill."Like" token, awarded by Boghog2."Like" token, awarded by Jonesey95."Like" token, awarded by BrownHairedGirl.

Description

Reported at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Search_totals_capped_at_10,000.

As part of the Elasticsearch 7 upgrade, search queries now "early stop" so they only return a total count up to the limit of 10k (which has always been the limit). This means that users can't see how many results total there are (the 10k search limit itself is not new).

On IRC @EBernhardson and @RKemper said it's possible to set a flag so it'll return the correct total count, but that there may be performance implications in doing so:

18:33:59 <+ebernhardson> yea, we can probably turn it back on (it's a boolean in the search query), and evaluate if it makes sense to save perf / latency in specific situations
18:34:11 <+ryankemper> Whether that use case is worth the extra CPU for us to change the behavior back is up for debate, but might not be a bad idea to flip that back so we're not surprising users with the behavior change
18:36:19 <+ryankemper> (Personally I'm leaning towards keeping the old count behavior but I definitely don't have a perfect grasp of how relevant the perf implications are)
18:36:43 <+ebernhardson> probably not worse than what we had yesterday without the early-stop :)

Event Timeline

Change 830953 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] Re-enable track_total_hits for elastic7

https://gerrit.wikimedia.org/r/830953

It would be great to have the totals restored. I find them very valuable

Change 830953 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Re-enable track_total_hits for elastic7

https://gerrit.wikimedia.org/r/830953

RKemper changed the task status from Open to In Progress.Sep 9 2022, 7:12 PM
RKemper updated the task description. (Show Details)

Change 831548 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@wmf/1.39.0-wmf.28] Re-enable track_total_hits for elastic7

https://gerrit.wikimedia.org/r/831548

Change 831548 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.39.0-wmf.28] Re-enable track_total_hits for elastic7

https://gerrit.wikimedia.org/r/831548

Mentioned in SAL (#wikimedia-operations) [2022-09-12T20:26:33Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:831548|Re-enable track_total_hits for elastic7 (T317374)]]

Mentioned in SAL (#wikimedia-operations) [2022-09-12T20:26:54Z] <samtar@deploy1002> samtar and ebernhardson: Backport for [[gerrit:831548|Re-enable track_total_hits for elastic7 (T317374)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-09-12T20:32:45Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:831548|Re-enable track_total_hits for elastic7 (T317374)]] (duration: 06m 12s)

Change 835142 had a related patch set uploaded (by DCausse; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@REL1_39] Re-enable track_total_hits for elastic7

https://gerrit.wikimedia.org/r/835142

Change 835142 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@REL1_39] Re-enable track_total_hits for elastic7

https://gerrit.wikimedia.org/r/835142