Context: We used to use the wb_terms table for entity search, and for this it has two extra columns, term_search_key and term_weight. However, since we now use CirrusSearch for entity search on Wikidata (and have done so for, I think, at least a year? a while, at least), those columns were unnecessarily bloating the database, so in T188993 we wiped them. (The columns still exist because having a different table schema on some wikis would be icky, but they’re never written to nor read from.)

The warning

Using wb_terms table for wbsearchentities API action but not using search-related fields of terms table. This results in degraded search experience, please enable the useTermsTableSearchFields setting.

indicates that, for some reason, we are now using wb_terms for searching again; since the term_search_key is not available, it falls back to term_text, which doesn’t work very well (making the search case sensitive is the most obvious effect, but various other normalizations are now also missing), hence the warning.

According to Logstash:

• This only happens on Wikidata, not on Test Wikidata. No idea why.
• This happened before the wmf.6 deployment as well, we just didn’t notice it. Therefore, I’m removing this as a train blocker, since it’s more likely to be due to a wmf-config change IMHO.

So Logstash says errors start happening around 1AM today (except for a single event about four hours earlier, apparently?), and SAL says T209402: A/B testing plan for wbsearchentities, context=item was deployed around that time, which sounds very related.

This comment at the bottom of that task sounds even more related (emphasis added):

In T209402#4779955, @EBernhardson wrote:

Test is shipped out. 10% of english item searches will use the newly tuned parameters. Another 10% will use the classic SQL search. Test is intended to run one week, to be disabled on Dec 4.

Change 476303 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Disable classic_entity wbsearchentities AB test

https://gerrit.wikimedia.org/r/476303

Change 476303 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable classic_entity wbsearchentities AB test

https://gerrit.wikimedia.org/r/476303

Mentioned in SAL (#wikimedia-operations) [2018-11-28T17:11:12Z] <thcipriani@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476303|Disable classic_entity wbsearchentities AB test]] T209402 T210618 (duration: 00m 55s)

Does it mean SQL-based search doesn't work now on Wikidata, or that we need to do some more setup to make it work? Would setting useTermsTableSearchFields be enough? Should we have it enabled in our mediawiki configs or disabling it also means we don't have proper index now and SQL search is unusable for us?

SQL search is intentionally disabled on Wikidata, and all the data that would back it (i. e. the term_search_key and term_weight columns) is not available. If for some reason we want to enable SQL search again, all that data needs to be reconstructed first, but the reason we disabled it in the first case is to reduce database bloat, because wb_terms is such a worryingly large table.

Basically, keeping the ability to fall back to SQL search costs us large amounts of database storage space, which doesn’t seem to be worth it when we really want to be using CirrusSearch anyways. We could discuss that decision, but it’s not a simple config change.

Got it, thanks. I guess then we have to give up on the idea to compare to SQL for the test in T209402: A/B testing plan for wbsearchentities, context=item and not do it in other tests. Not a big deal, I was just not aware it's not an option anymore.