Page MenuHomePhabricator

Migrate search integration of ArticlePlaceholder to elastic
Closed, ResolvedPublic8 Estimated Story Points


As * I want to kill the wb_terms table so that it will stop exploding


ArticlePlaceholder currently uses the wb_terms table for search.
The use of wb_terms has caused us many issues over the years, causing an outage[[ | rather recently (May 2018) ]].
As a result the ArticlePlaceholder search integration is currently disabled (it will be re enabled with T195751).
In order to re enable the feature and ensure stability and reliability we should first migrate the feature away from wb_terms and instead use elasticsearch.

Using elasticsearch directly in PHP poses some difficulties as all of the elastic & search code is in Repo and ArticlePlacholder only has access to Client.
During a discussion with @Legoktm, @Aleksey_WMDE & I Wikimania this year we discussed the idea of calling the repo wbsearchentities API internally from the client, this is probably the best path forward.

Impact & Priority

The feature should probably not be enabled again until this task is resolved, hence T195751 is blocked by this.
As for the priority of T195751 (turning the feature back on) that is down to @Lydia_Pintscher.
When writing this description this task was marked as "High" so I'll leave it as that for now (2018-08-19)


Make ArticlePlaceholder use elasticsearch instead of the wb_terms table by calling the repo wbsearchentities API module.

Acceptance criteria

  • ArticlePlaceholder on longer uses the wb_terms table for anything
  • ArticlePlaceholder uses wikidata search via the wbsearchentities API module
  • Calls to the API module should have a short timeout and should fail gracefully & log success / failure rates


This would be replacing the search interactor with a new interactor / service that would call the API instead:

Event Timeline

Addshore renamed this task from Migrate search integration of article place holder to elastic to Migrate search integration of ArticlePlaceholder to elastic.Aug 19 2018, 10:47 AM
Addshore added a project: wdwb-tech.
Addshore updated the task description. (Show Details)
Addshore moved this task from Inbox to Goals on the wdwb-tech board.
Addshore added a subscriber: Legoktm.

Change 462760 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/ArticlePlaceholder@master] Add API-based search integration

Change 462760 merged by jenkins-bot:
[mediawiki/extensions/ArticlePlaceholder@master] Add API-based search integration

We need to enable it first and then change the repo url (in a test wiki) and with that, we should get different results. That's the easiest way we can test it.

Addshore lowered the priority of this task from High to Medium.Oct 9 2018, 1:44 PM

Will this work with the API:Search i.e. if I add a language that has Articleplace holder activated then I can search it

We verified this on the beta cluster.
It will be deployed as part of T195751