Support multi-project indices
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	EBernhardson
	Jul 6 2016, 6:25 PM

Description

To support the discovery of more content add support for including multiple projects of the same language in the same search index.

The initial production rollout should support showing users results only from the project they are searching on, as it is today, allowing us to start running tests where we integrate result sets from other projects.

Related Objects
Search...

Status	Assigned	Task
Resolved	debt	T139498 Support multi-project indices
Resolved	EBernhardson	T139496 Centralize document id and namespace generation
Declined	None	T139499 Adjust the process of building indices to allow multiple build steps before promoting to production.
Resolved	EBernhardson	T139500 Add a field to elasticsearch documents identifying the project it is from
Declined	None	T139502 Update popularity score import to handle multi project indices
Declined	None	T139504 Limit completion suggester building to single project
Declined	None	T139505 Evaluate cross-project query boosting
Resolved	EBernhardson	T140434 Use an explicit index prefix instead of wfWikId()
Declined	None	T140435 Create a way to iterate over wikis that share the same search index
Declined	None	T140436 Add a maintenance script flag to indicate the user knows they are operating on a multi-wiki index
Resolved	dcausse	T141033 Centralize conversion of an elasticsearch hit into a Title object
Declined	None	T141045 Audit usage of filters for wiki-specific items (namespaces, templates, categories, etc) as make boolean queries where necessary
Resolved	CKoerner_WMF	T137312 Work with communities to learn how inter-wiki search might work

Event Timeline

EBernhardson created this task.Jul 6 2016, 6:25 PM

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 6 2016, 6:25 PM

EBernhardson added a subtask: T139496: Centralize document id and namespace generation.Jul 6 2016, 6:26 PM

EBernhardson created subtask T139499: Adjust the process of building indices to allow multiple build steps before promoting to production..Jul 6 2016, 6:30 PM

EBernhardson created subtask T139500: Add a field to elasticsearch documents identifying the project it is from.Jul 6 2016, 6:33 PM

EBernhardson added a project: CirrusSearch.

Restricted Application added projects: Discovery-ARCHIVED, Discovery-Search. · View Herald TranscriptJul 6 2016, 6:33 PM

EBernhardson created subtask T139502: Update popularity score import to handle multi project indices.Jul 6 2016, 6:36 PM

EBernhardson created subtask T139504: Limit completion suggester building to single project.Jul 6 2016, 6:38 PM

EBernhardson created subtask T139505: Evaluate cross-project query boosting.

EBernhardson updated the task description. (Show Details)Jul 6 2016, 6:42 PM

debt triaged this task as Medium priority.Jul 6 2016, 9:16 PM

Danny_B renamed this task from [EPIC] Support multi-project indices to Support multi-project indices.Jul 6 2016, 10:44 PM

Danny_B added a project: Epic.

EBernhardson added a subtask: T140434: Use an explicit index prefix instead of wfWikId().Jul 14 2016, 11:37 PM

EBernhardson added a subtask: T140435: Create a way to iterate over wikis that share the same search index.

EBernhardson added a subtask: T140436: Add a maintenance script flag to indicate the user knows they are operating on a multi-wiki index.Jul 14 2016, 11:40 PM

debt moved this task from needs triage to This Quarter on the Discovery-Search board.Jul 19 2016, 10:24 PM

EBernhardson created subtask T141033: Centralize conversion of an elasticsearch hit into a Title object.Jul 21 2016, 6:47 PM

EBernhardson created subtask T141045: Audit usage of filters for wiki-specific items (namespaces, templates, categories, etc) as make boolean queries where necessary.Jul 21 2016, 8:53 PM

One issue I can think of right now: as we are making content model more and more rich, we may have additional fields for different content models in each wiki. This can be Flow, structured data, extensions, etc. We need to ensure these fields do not clash (i.e. we don't have two extensions which try to use field "my_data" with conflicting definitions) or somehow find way of namespacing them.

debt closed subtask T139500: Add a field to elasticsearch documents identifying the project it is from as Resolved.Aug 8 2016, 4:58 PM

debt closed subtask T140434: Use an explicit index prefix instead of wfWikId() as Resolved.

CKoerner_WMF added a subtask: T137312: Work with communities to learn how inter-wiki search might work.Aug 12 2016, 7:51 PM

debt closed subtask T139496: Centralize document id and namespace generation as Resolved.Aug 26 2016, 4:28 PM

CKoerner_WMF closed subtask T137312: Work with communities to learn how inter-wiki search might work as Resolved.Sep 27 2016, 8:45 PM

Liuxinyu970226 subscribed.Oct 1 2016, 7:44 AM

• Deskana closed subtask T141033: Centralize conversion of an elasticsearch hit into a Title object as Resolved.Dec 9 2016, 3:18 PM

After a full evaluation of our options, and some testing, we came to the conclusion that multi-project indices is a premature optimization. While it may be more performant, it seems the extra complexity is not worth it. Should we decline the series of open tasks?

Closing this ticket as resolved and I'll close out the accompanying subtasks, based on @EBernhardson's comment above and checking with @TJones to be sure he's onboard. This ticket was written up to solve a problem that ended up not being a problem. :)