Page MenuHomePhabricator

Change loading of cross-wiki configuration to use cirrus-dump-config api call
Closed, ResolvedPublic

Description

While reviewing a patch that desired to load cross-wiki configuration for sister search it occured to me we could use an api call rather than the forking that we do today. We already have an endpoint for this, although the whitelist would probably need to be updated.

Proof of concept: P4820

Testing this in mwrepl on an app server takes 50-100ms to load all the sister wikis. Still not great but better than our current solution. Even if we don't use it for sister wikis and only for the textcat interwiki search, this might be a reasonable solution

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2017, 4:24 PM

I'm not sure if this is the right way forward, but seems possible?

Restricted Application added projects: Discovery, Discovery-Search. · View Herald TranscriptJan 27 2017, 4:25 PM

But doesn't the forking solution also load config vars that are not available in cirrus-dump-config?

It certainly does, including non-cirrus values. We could probably change the way we generate cirrus-dump-config to output everything we need though

Deskana triaged this task as Low priority.Feb 2 2017, 11:23 PM
Deskana added a subscriber: Deskana.

Not urgent now. We can bump priority if it becomes urgent later.

This would prevent errors like T160569

Deskana raised the priority of this task from Low to Normal.Mar 16 2017, 10:13 PM
Deskana moved this task from later on... to Current work on the Discovery-Search board.
debt added a subscriber: debt.Mar 21 2017, 5:17 PM

We hope to get to this soon, keeping in the backlog on the sprint board for now.

This should also be useful for general multimedia search, specifically commons defines some template boosts which seem to improve at least some rankings. When we do multimedia search from a non-commons wiki though these template boosts aren't taken into account. We could probably return them somehow as part of this ticket and improve wiki-local multimedia search.

Change 366609 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Move configvars whitelist into Api/ConfigDump

https://gerrit.wikimedia.org/r/366609

Change 366609 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Move configvars whitelist into Api/ConfigDump

https://gerrit.wikimedia.org/r/366609

This should also be useful for general multimedia search, specifically commons defines some template boosts which seem to improve at least some rankings. When we do multimedia search from a non-commons wiki though these template boosts aren't taken into account. We could probably return them somehow as part of this ticket and improve wiki-local multimedia search.

Worth noting we took a different direction for commons search, basically calling out those templates in the config for all wikis rather than trying to source the commons config.

Change 377773 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Load interwiki configs via cirrus config dump api

https://gerrit.wikimedia.org/r/377773

Change 377773 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Load interwiki configs via cirrus config dump api

https://gerrit.wikimedia.org/r/377773

debt closed this task as Resolved.Oct 6 2017, 7:42 PM