Change loading of cross-wiki configuration to use cirrus-dump-config api call
Closed, ResolvedPublic
Actions

Description

While reviewing a patch that desired to load cross-wiki configuration for sister search it occured to me we could use an api call rather than the forking that we do today. We already have an endpoint for this, although the whitelist would probably need to be updated.

Proof of concept: P4820

Testing this in mwrepl on an app server takes 50-100ms to load all the sister wikis. Still not great but better than our current solution. Even if we don't use it for sister wikis and only for the textcat interwiki search, this might be a reasonable solution

Details

	Project	Branch	Lines +/-	Subject
	mediawiki/extensions/CirrusSearch	master	+803 -116	Load interwiki configs via cirrus config dump api
	mediawiki/extensions/CirrusSearch	master	+163 -142	Move configvars whitelist into Api/ConfigDump

Customize query in gerrit

Related Objects

Mentioned In: T173692: wgCirrusSearchInterleaveConfig is not set
T151935: Cross-wiki search results: making sure the results are relevant from sister projects
Mentioned Here: T160569: Variable wgCirrusSearchCompletionGeoContextSettings is not set in getConfiguration.php on line 105
P4820 example multi-request for cirrus config parameters

Event Timeline

EBernhardson created this task.Jan 27 2017, 4:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2017, 4:24 PM

I'm not sure if this is the right way forward, but seems possible?

EBernhardson added a project: CirrusSearch.Jan 27 2017, 4:25 PM

Restricted Application added projects: Discovery-ARCHIVED, Discovery-Search. · View Herald TranscriptJan 27 2017, 4:25 PM

But doesn't the forking solution also load config vars that are not available in cirrus-dump-config?

It certainly does, including non-cirrus values. We could probably change the way we generate cirrus-dump-config to output everything we need though

EBernhardson mentioned this in T151935: Cross-wiki search results: making sure the results are relevant from sister projects.Jan 31 2017, 7:14 PM

Not urgent now. We can bump priority if it becomes urgent later.

Deskana moved this task from needs triage to search-icebox on the Discovery-Search board.Feb 2 2017, 11:23 PM

This would prevent errors like T160569

Deskana raised the priority of this task from Low to Medium.Mar 16 2017, 10:13 PM

Deskana moved this task from search-icebox to Current work on the Discovery-Search board.

Deskana edited projects, added Discovery-Search (Current work); removed Discovery-Search.

We hope to get to this soon, keeping in the backlog on the sprint board for now.

This should also be useful for general multimedia search, specifically commons defines some template boosts which seem to improve at least some rankings. When we do multimedia search from a non-commons wiki though these template boosts aren't taken into account. We could probably return them somehow as part of this ticket and improve wiki-local multimedia search.

dcausse claimed this task.Jul 20 2017, 12:49 PM

dcausse moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.

Change 366609 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Move configvars whitelist into Api/ConfigDump

https://gerrit.wikimedia.org/r/366609

gerritbot added a project: Patch-For-Review.Jul 20 2017, 6:28 PM

Change 366609 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Move configvars whitelist into Api/ConfigDump

https://gerrit.wikimedia.org/r/366609

In T156497#3190409, @EBernhardson wrote:

This should also be useful for general multimedia search, specifically commons defines some template boosts which seem to improve at least some rankings. When we do multimedia search from a non-commons wiki though these template boosts aren't taken into account. We could probably return them somehow as part of this ticket and improve wiki-local multimedia search.

Worth noting we took a different direction for commons search, basically calling out those templates in the config for all wikis rather than trying to source the commons config.

ReleaseTaggerBot added a project: MW-1.30-release-notes (WMF-deploy-2017-07-25_(1.30.0-wmf.11)).Jul 21 2017, 2:00 AM

EBernhardson mentioned this in T173692: wgCirrusSearchInterleaveConfig is not set.Aug 21 2017, 4:22 PM

Change 377773 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Load interwiki configs via cirrus config dump api

https://gerrit.wikimedia.org/r/377773

gerritbot added a project: Patch-For-Review.Sep 13 2017, 2:25 PM

dcausse moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Sep 13 2017, 2:38 PM

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Oct 2 2017, 4:03 PM

Change 377773 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Load interwiki configs via cirrus config dump api

https://gerrit.wikimedia.org/r/377773

debt closed this task as Resolved.Oct 6 2017, 7:42 PM