Page MenuHomePhabricator

Restore cross-wiki (interwiki) search across all public Wikimedia wikis
Open, NormalPublic

Description

We apparently already have the ability to do cross-wiki (or interwiki) searches using CirrusSearch/Elasticsearch, but the functionality has not yet been enabled everywhere. This task is to enable cross-wiki (or interwiki) search on all public Wikimedia wikis.

Event Timeline

MZMcBride updated the task description. (Show Details)
MZMcBride raised the priority of this task from to Needs Triage.
MZMcBride added a subscriber: MZMcBride.
Restricted Application added a project: Discovery. · View Herald TranscriptAug 22 2015, 9:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Nemo_bis triaged this task as Normal priority.Aug 22 2015, 9:30 PM
Nemo_bis edited projects, added Wikimedia-Search; removed MediaWiki-Search.
Nemo_bis set Security to None.
Nemo_bis renamed this task from Enable cross-wiki (interwiki) search across all public Wikimedia wikis to Restore cross-wiki (interwiki) search across all public Wikimedia wikis.
Nemo_bis added a subscriber: Nemo_bis.
Glaisher added a subscriber: Glaisher.
Deskana lowered the priority of this task from Normal to Lowest.Dec 29 2015, 11:13 PM
Deskana added a subscriber: Deskana.

Restoring inter-wiki search in its existing state (i.e. like you see it on the Italian Wikipedia) is low priority for Discovery. We are working on a number of inter-wiki, multi-language search projects, however.

Deskana moved this task from Needs triage to Search on the Discovery board.Dec 29 2015, 11:13 PM
Nemo_bis raised the priority of this task from Lowest to Normal.Dec 30 2015, 7:38 AM

This bug is a regression and as such fixing it is normal priority. Others can work on this.

Restricted Application added subscribers: JEumerus, Matanya. · View Herald TranscriptDec 30 2015, 7:38 AM
Nemo_bis added a project: Regression.
debt added a subscriber: debt.Feb 16 2016, 4:49 PM
Elitre added a subscriber: Elitre.Jun 8 2016, 2:30 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptJun 8 2016, 2:30 PM

It would be great if someone could clarify whether or not this is actually a regression. The title implies it is, as does one comment. But the description seems to say that nothing has changed, and instead that this is a request to extend existing functionality to more contexts.

i'm not familiar with interwiki search ever being enabled everywhere. The current implementation is a hack and our infrastructure cannot support enabling it everywhere. The tl/dr of the current implementation is it walks through a list of wiki's to query and issues a query for each individual wiki. Enabling the current interwiki search implementation would basically explode both response times and server load in the search cluster.

Nemo_bis added a comment.EditedJun 10 2016, 6:36 AM

i'm not familiar with interwiki search ever being enabled everywhere.

It was in 2009, see T46420. Users have always liked the feature a lot.

I see that searching all wikis can be a huge performance issue.
As an alternative I would suggest to add interwiki links to the search result page that simply links to search result pages of other wikis without issuing a query on these wikis.

debt added a comment.Jun 27 2017, 6:27 PM

Hi,

Just to follow-up on this ticket, we launched sister project snippets (T145917) on the search results page (T162276) on June 15, 2017 and it is now available on all Wikipedias. However, enwiki has a different interface (fewer sister projects) as they had requested via an RfC (T162276#3278689) discussion.

Due to unforeseen issues, we don't have event logging publicly available yet, but we're working on getting the new dashboard statistics (T164854) into production soon.

Yesterday, we did a query of clickthrough traffic across all Wikipedias to get a simple snapshot idea of the usage of the sister projects (from the search results page) before and after the production release:

Yesterday, we did a query of clickthrough traffic across all Wikipedias to get a simple snapshot idea of the usage of the sister projects (from the search results page) before and after the production release:

This is very useful! Wiktionary is an obvious first, but Wikinews is doing surprisingly well (compared to its traffic: good traffic for Wikisource, Wikiquote and Wikibooks is unsurpising). This suggests that cross-wiki search results are indeed covering queries which are inappropriate for Wikipedia (recent news, word definitions).

The absolute numbers are still low, but they'll need to be monitored in the longer term and I expect most impact to come from a self-sustaining virtuous circle as users learn about these additional resources.