FYI it looks like data isn't being written to the search index

Captions should be getting written to opening_text. See below - the cirrus data is old

https://commons.wikimedia.beta.wmflabs.org/wiki/File:Photo_on_06-12-2018_at_10.20.jpg
https://commons.wikimedia.beta.wmflabs.org/wiki/File:Photo_on_06-12-2018_at_10.20.jpg?action=cirrusDump

Correction - never mind the comment above, eventually the content showed up in the index, so that's not the problem

"An error has occurred while searching" usually means not missing data in index (in that case search would just return nothing, usually) but some kind of error returned by Elastic. I'd check the debug logs - they usually have more information about what exactly Elastic is unhappy about.

Pinging @EBjune to see if we can borrow someone with more expertise in Elastic to help us debug this, per Stas's mention above. :)

If you give me access to the instance, I could take a quick look.

In T212138#4860903, @Smalyshev wrote:

If you give me access to the instance, I could take a quick look.

This is on Beta Cluster (deployment-prep), so it should show up in the logstash system somewhere (but I can't see it), though you appear to be a project owner there so can just sudo in…?

This is what I am getting in the log:

2019-01-07 23:47:53 [XDPlKawQBHcAAEAR02gAAABS] deployment-mediawiki-07 commonswiki 1.33.0-alpha CirrusSearch WARNING: Search backend error during full_text search for 'duck' after 12: parse_exception: parse_exception: Field [sitelink_count] does not exist in mappings {"queryType":"full_text","tookMs":12,"query":"duck","limit":21,"suggestion":null,"syntax":["full_text","entity_full_text","full_text_simple_match"],"index":"commonswiki","hitsOffset":0,"error_message":"parse_exception: parse_exception: Field [sitelink_count] does not exist in mappings"}

Looks like somehow a field is missing - probably needs a manual reindex. I don't want to mess anything up so I won't touch anything for now but if you want, I can try to reindex and see what happens.

I did a reindex and the error is gone. Please see if any other help is needed.

This is great, thanks Stas!

@Ramsey-WMF – Note that this now "correctly" shows the error "A warning has occurred while searching: Mixing entity and article namespaces in search is currently not supported. Only entity namespaces will be searched.", i.e. wikibase-search-namespace-mix. If you weren't aware of this limitation, you should file a ticket.

There's already a ticket about it - T194968: Enable search in all wikidata namespaces combined. And T204813: Allow looking for items and lexemes namespaces together by default for smaller-scope more-likely-to-happen-anytime-soon one.

All right. Thanks everyone for the input so far.

Sooooo, I have two questions.

a.) I want to confirm that if federation isn't enabled, this isn't a problem. Search worked as expected back when we were only testing captions; are we sure that's still the case so we can at least launch captions without worrying about this (for now)?

b.) What's the likelihood of us getting T194968 (or something similar to the solution for T204813) ready by end of January? :)

@Jdforrester-WMF afaics that error is something screwy with namespaces on beta

Searching for 'cat' on beta gives me this url https://commons.wikimedia.beta.wmflabs.org/w/index.php?search=cat&title=Special%3ASearch&profile=advanced&fulltext=1&advancedSearch-current=%7B%22namespaces%22%3A%5B0%2C6%2C12%2C14%2C100%2C106%5D%7D&ns0=1&ns6=1&ns12=1&ns14=1&ns100=1&ns106=1

If you remove ns0=1 then it works without the warning, which suggests that ns 0 is somehow set up as an entity namespace

In T212138#4861204, @Smalyshev wrote:

I did a reindex and the error is gone. Please see if any other help is needed.

Hmm. This error now occurs on TestCommons, which is configured with captions-only (no Wikibase federation): https://test-commons.wikimedia.org/w/index.php?search=&search=fish&title=Special%3ASearch&go=Go

Presumably this means that it will occur on real Commons as soon as we switch SDC on, which is a really big problem (as I suspect a manual rebuild there takes a bit longer than a few seconds).

These 2 queries are rather different on commons and testcommons. I wonder why?

https://test-commons.wikimedia.org/w/index.php?search=&search=salmonid&title=Special%3ASearch&go=Go&cirrusDumpQuery

https://commons.wikimedia.org/w/index.php?search=salmonid&title=Special%3ASearch&go=Go&cirrusDumpQuery

The testcommons one includes the sitelink_count field that caused problems on beta, the production commons one doesn't

If you enable Wikibase searching (no matter federated or not, I'm not familiar with test-commons setup so tell me if I'm talking nonsense here) then it should have the fields in the index, otherwise you'd get errors.

I have no idea why it considers namespace 0 to be entity one, that probably needs to be debugged - that should be configurable, so maybe some config is missing, but maybe also a bug.

Now for rolling it out - since search settings are configurable, we could make it so that the initial config does not use any fields that aren't in the default index, but it would be better to just not enable searching Wikibase namespaces until we indexed them. Not sure how easy it's to do dpeloyment-wise, maybe we'd need some code to ensure it?

What's the likelihood of us getting T194968 (or something similar to the solution for T204813) ready by end of January? :)

With the resources and plans we have now - quite unlikely. We could talk about reprioritizing, but even if we'd drop everything (which we can't) and start working on it right now - we have 2.5 weeks in January left, given that the last week is All Hands, and this is not something that I think we can realistically start & finish in that timeframe. Unless somebody has some bright idea I've missed how to do it easier than I think (which I am all for it then :)

Presumably this means that it will occur on real Commons as soon as we switch SDC on, which is a really big problem (as I suspect a manual rebuild there takes a bit longer than a few seconds).

Remembering the Wikidata rollout, here's how we did it:

• Make an option to enable Wikibase search, but keep it off by default. Deploy the code. ElasticSearch is fine using unknown fields in indexing, IIRC, so even though indexing wouldn't work properly yet for Wikibase, no problems should follow.
• Do an in-place reindex for the wiki. This should create all mappings necessary for new code.
• If you have any Wikibase data (or Wikibase data derived from existing data) at this point, you'll need to reindex the pages with this data using full reindex. Pages without Wikibase data won't need full reindex.
• At this point you can change the configs to enable Wikibase search. If you want to be cautions, make the option able to be triggered by request parameter, and check with that first, before enabling it in the permanent config.

OK, I think I've found the cause of this, which is a config mistake which should be now fixed in production. Will confirm.

Yeah, the cirrusDumpQuery is now the same. Sorry for the confusion!

... and searching now works on testcommons! So - are we good to go?