vagrant wikibase cirrus role not working / should set $wgWBCSUseCirrus = true; to enable
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Mvolz
	Jul 19 2019, 10:53 AM

Description

I've enabled the vagrant wikibase cirrus role, however I'm still getting empty results for srsearch queries, i.e. wbhasstatement gives me:

{
    "batchcomplete": "",
    "query": {
        "searchinfo": {
            "totalhits": 0
        },
        "search": []
    }
}

I've tried reindexing from CirrusSearch, which succeeds, but doesn't help:

> mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php
[                wiki] Indexed 3 pages ending at 3 at 3/second
Indexed a total of 3 pages at 3/second

The puppet file didn't look like it enabled it, so I tried enabling it according to the docs, but that didn't work either:

http://wikidata.wiki.local.wmftest.net:8080/w/api.php?action=query&list=search&srsearch=haswbstatement:P7=7&useCirrus=1

This results in

"main": {
    "*": "Unrecognized parameter: useCirrus."
},

I've also tried manually enabling in in my localsettings, i.e.

$wgWBCSUseCirrus = true;

This results in

{
    "batchcomplete": "",
    "query": {
        "searchinfo": {
            "totalhits": 0
        },
        "search": []
    }
}

Even after reprovisioning, reindex using mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php

Details

	Subject	Repo	Branch	Lines +/-
	[mediainfo] Tell cirrus to index P1 and P2 as statements	mediawiki/vagrant	master	+2 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Smalyshev	T190022 Separate the CirrusSearch/Elastic-specific code from Wikibase code base
Resolved		Smalyshev	T215751 Make vagrant support WikibaseCirrusSearch
Resolved	BUG REPORT	dcausse	T228503 vagrant wikibase cirrus role not working / should set $wgWBCSUseCirrus = true; to enable

Event Timeline

Mvolz created this task.Jul 19 2019, 10:53 AM

Mvolz updated the task description. (Show Details)

Tgr subscribed.Jul 19 2019, 11:04 AM

Smalyshev triaged this task as Medium priority.Jul 31 2019, 8:10 PM

Smalyshev added a project: Discovery-Search (Current work).

Smalyshev moved this task from Backlog to Doing on the User-Smalyshev board.

I tried to install wikibasecirrussearch with vagrant and it seems to work for me, after manually enabling $wgWBCSUseCirrus = true; (that part is not automatic yet). Maybe some other configurations are interfering? Are there any local configs and where do you configure $wgWBCSUseCirrus = true;? Did you check that you have latest master checkout of all modules (CirrusSearch, Wikibase, WikibaseCirrusSearch)?

Smalyshev moved this task from Doing to In review on the User-Smalyshev board.Aug 1 2019, 4:42 AM

Smalyshev moved this task from In review to Waiting/Blocked on the User-Smalyshev board.

In T228503#5383302, @Smalyshev wrote:

I tried to install wikibasecirrussearch with vagrant and it seems to work for me, after manually enabling $wgWBCSUseCirrus = true; (that part is not automatic yet). Maybe some other configurations are interfering? Are there any local configs and where do you configure $wgWBCSUseCirrus = true;? Did you check that you have latest master checkout of all modules (CirrusSearch, Wikibase, WikibaseCirrusSearch)?

On update I'm no longer getting the error with $wgWBCSUseCirrus = true, but I'm still not getting any results through with haswbstatement queries, even after reprovisioning- any further debugging tips?

Mvolz updated the task description. (Show Details)Aug 1 2019, 12:59 PM

Mvolz renamed this task from vagrant wikibase cirrus role not working to vagrant wikibase cirrus role not working / should set $wgWBCSUseCirrus = true; to enable.Aug 2 2019, 8:38 AM

Smalyshev removed a project: User-Smalyshev.Aug 7 2019, 6:59 AM

Tgr mentioned this in T228428: Vagrant role for the MachineVision extension.Aug 7 2019, 11:25 AM

Tgr mentioned this in T230011: MediaWiki-Vagrant: mediainfo role does not work.Aug 7 2019, 12:12 PM

Gehel assigned this task to dcausse.Aug 27 2019, 5:29 PM

The searchIndexProperties wikibase config needs to be populated as well.

What I did (for the record):

fetch a fresh vagrant
enable wikibasecirrussearch
provision (ran into T231534)
ran vagrant provision a second time
created /vagrant/settings.d/wikis/wikidatawiki/settings.d/20-cirrus.php to set $wgWBCSUseCirrus = true;
edited /vagrant/mediawiki/extensions/repo/config/Wikibase.default.php and set 'searchIndexProperties' => ['P1'] (there are probably better ways to customize wikibase in vagrant)
reindexed wikidatawiki: mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki wikidatawiki --reindexAndRemoveOk --indexIdentifier now
ran into GC issues (T211362)
edited /etc/elasticsearch/jvm.options and set -Xmx512
deleted partially created indices: curl -XDELETE http://localhost:9200/wikidatawiki*
created the indices: mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki wikidatawiki
populated the index: mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki wikidatawiki
created P1 and an entity using it, searched for haswbstatement:P1, haswbstatement:* both returned Q1

I'll increase the heap as suggested in T211362.

As for this particular ticket I'm not sure what to do, I'm tentatively moving it to done, @Mvolz please let me know if setting searchIndexProperties did solve the problem.

Thanks for lookig into it! These should be folded into the Vagrant role settings. I can do that in a few weeks; if you want to do it before that, the mediainfo role has an example of how to create Wikibase properties/entities (that should probably be moved to the wikidata and wikibase_repo roles so it's always available), JVM options are in puppet/modules/elasticsearch/files/jvm.options and most of the configuration should probably happen via the mediawiki::extension resource in the wikibasecirrussearch role. The reindexing should be in theory handled by the cirrussearch role if the other settings are correct.

dcausse moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.Aug 29 2019, 3:18 PM

dcausse moved this task from Needs review to Incoming on the Discovery-Search (Current work) board.

Change 533246 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/vagrant@master] [mediainfo] Tell cirrus to index P1 and P2 as statements

https://gerrit.wikimedia.org/r/533246

gerritbot added a project: Patch-For-Review.Aug 29 2019, 3:36 PM

@Tgr thanks for the info!
I've uploaded a patch but I attached the config listing the properties to index under mediainfo, reason is that it's mediainfo that creates these properties. Since we have to create some properties I'm not entirely sure what to do in wikibasecirrussearch...
As for $wgWBCSUseCirrus = true I'm still unsure but IIRC @Smalyshev had some reasons not to set it to true by default.