Create an API for fulltext search across all wikis
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	CKoerner_WMF
	Nov 21 2017, 4:46 PM

Description

Summary
As requested, "an API for that across-all-wikis fulltext search yet? Maybe just on Toolforge?"

https://twitter.com/MagnusManske/status/928270803833155586

Description
Provide an API that would allow users to perform a full text search across all Wikimedia wikis. Due to the cost, perhaps extremely rate limited to limited to requests for access.

When asked about the suggestion, the a search engineers said the following:

"we asked for a budget to do it on toolforge. It wasn't approved.
well, it was approved in the sense of "we wont give you money for it, but if theres any left over then maybe"
in theory we could setup a rediculously rate limited api for it without much work against the prod clusters.
(and also limiting keywords agressively, like no regex)"

Related Objects

Mentioned Here: T145918: [EPIC] Backend: cross-wiki search results functionality

Event Timeline

CKoerner_WMF created this task.Nov 21 2017, 4:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 21 2017, 4:46 PM

Magnus subscribed.Nov 21 2017, 4:48 PM

For previous, related, non-API work, see T145918: [EPIC] Backend: cross-wiki search results functionality

If this was in production, it'd mean that we're searching across all shards which is incredibly expensive. We need to take into account operational concerns, available servers, money, etc to see if we can take this on and then how many people would actually use this.

Liuxinyu970226 subscribed.Jun 20 2018, 10:58 PM

CKoerner_WMF unsubscribed.Dec 12 2019, 4:20 PM

Closing out low/est priority tasks over 6 months old with no activity within last 6 months in order to clean out the backlog of tickets we will not be addressing in the near term. Please feel free to reopen if you think a ticket is important, but bare in mind that given current priorities and resourcing, it is unlikely for the Search team to pick up these tasks for the indefinite future. We hope that the requested changes have either been addressed by or made irrelevant by work the team has done or is doing -- e.g. upgrading Elasticsearch to a newer version will solve various ES-related problems -- or will be subsumed by future work in a more generalized way.

Re-opening tasks and removing from team workboard per IRC feedback given yesterday and discussion with MPham.

• MPhamWMF added a project: Discovery-ARCHIVED.Aug 2 2022, 1:39 PM

Cloudelastic is covering this requirement.

Create an API for fulltext search across all wikisClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Create an API for fulltext search across all wikis
Closed, ResolvedPublic
Actions