Maniphest T317381

Reduction in helpfulness and quantity of autocomplete search results
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

Visit https://en.wikipedia.org/
In the search box, start typing Uyghur genocide

What happens?:

Uyghur genocide article is not suggested
No articles starting with Uyghur are suggested
No autocomplete results are offered at all for "Uygh", "Uyghu", "Uyghur g", etc.

What should have happened instead?:

Articles such as Uyghur genocide should be suggested

Software version (skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

Originally reported at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Weird_and_unhelpful_autocompleted_search_results

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		None	T248925 Make MediaWiki release tarball compatible with PHP 8.0
Resolved		Jdforrester-WMF	T300463 Make PHP 8.0 voting on MW master
Resolved		None	T283275 Make MW master tests pass on PHP 8.0
Resolved		Reedy	T268861 CirrusSearch uses Elastica's Match class
Resolved		Reedy	T268863 Translate uses Elastica's Match class
Resolved		matthiasmullie	T268866 WikibaseMediaInfo uses Elastica's Match class
Invalid		None	T268864 WikibaseCirrusSearch uses Elastica's Match class
Resolved		Reedy	T268865 WikibaseLexemeCirrusSearch uses Elastica's Match class
Resolved		EBernhardson	T271777 Bump rufin/elastica (and related libraries) to versions that support PHP 8.0
Resolved		Gehel	T263142 [EPIC] Upgrade Elasticsearch to version 7.10
Resolved		EBernhardson	T308676 Elasticsearch 7.10.2 rollout plan
Resolved	BUG REPORT	EBernhardson	T317381 Reduction in helpfulness and quantity of autocomplete search results

Event Timeline

Novem_Linguae created this task.Sep 9 2022, 2:30 AM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptSep 9 2022, 2:30 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Novem_Linguae updated the task description. (Show Details)Sep 9 2022, 2:31 AM

Novem_Linguae updated the task description. (Show Details)

cross-referencing https://gerrit.wikimedia.org/r/830962 and:

<+logmsgbot> !log ebernhardson@deploy1002 Synchronized wmf-config/InitialiseSettings.php: cirrus: Switch all wikis from completion suggester to prefix search, yesterdays completion index builds in codfw weren't all succesfull and users are getting incomplete results (duration: 04m 01s)

The commit message mostly says what went wrong here, this is related to the switchover of major versions of elasticsearch, the database that powers search across the wikis (and when elasticsearch says major versions are for breaking changes, they mean it!)

The completion suggester build process for wmf.27 was only compatible
with es6, wmf.28 with es7. That means the codfw cluster currently has
a variety of bad completion suggester indices.  Switch back to the
strict prefix search until those have all been rebuilt.

We probably should have noticed this in our integration testing, but sadly it was missed. For now with the above patch deployed wikis are using the old strict-prefix search which isn't forgiving of typos, but does contain all known pages and redirects. The indices that were built yesterday in codfw (the cluster we've migrated to 7 and is now serving all traffic) for autocomplete are incomplete and missing many things. This should resolve itself in the coming hours and we can turn the completion suggester, which is much more forgiving of typo's, back on.

From #wikimedia-search:

[02:13:25] <legoktm>	 ebernhardson: if you're still around, https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Weird_and_unhelpful_autocompleted_search_results
[04:21:55] <ebernhardson>	 odd, looking
[04:30:52] <ebernhardson>	 enwki_titlesuggest is suspiciously small, 24k docs
[04:45:33] <ebernhardson>	 titlesuggest build in eqiad totally failed, started at 2:33:42 and finished at 2:34:20, compared to yesterday it started at 3:09:45 and finished at 4:05:44. probably due to it expecting es7 but talking to an es6 cluster (but noone is querying eqiad, not a huge problem)
[04:49:49] <ebernhardson>	 codfw less clear,  enwiki was still rebuilding when issues were reported, that should have still been on the old index before the new one in promoted
[04:50:30] <ebernhardson>	 other indices seem plausible, dewiki has finished it's rebuild and has what looks like a normal amount of docs
[04:56:12] <ebernhardson>	 oh! yesterdays enwiki build failed...it seems whats happening is the build process in wmf.27 is only valid against es6, wmf.28 is only valid against es7. We should probably have done something weird like stopped the rebuilds and accepted old completion indices during the switchover, but it shuld all fix itself over the next couple hours 
[04:56:41] <ebernhardson>	 (and should have realized this index build had to be compatible in both directions, not something we thought about...)
[05:01:11] <ebernhardson>	 I'll switch $wgCirrusSearchUseCompletionSuggester to build, which will use prefix search everywhere. Around 8 or 9 am UTC rebuilds should be complete everywhere and can revert it
[05:04:50] <ebernhardson>	 (and the logs are now a bit spammed up with index_not_found errors on enwiki's codfw build today because i restarted it when it probably wasn't necessary)
[05:10:37] <ebernhardson>	 one potential problem, autocomplete is cached for 3 hours :S so logged out users can still get bad results for awhile

EBernhardson added a parent task: T308676: Elasticsearch 7.10.2 rollout plan.Sep 9 2022, 5:29 AM

Mentioned in SAL (#wikimedia-operations) [2022-09-09T14:43:29Z] <dcausse@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T317381: Revert "Disable CirrusSearch completion suggester" (duration: 03m 57s)