Page MenuHomePhabricator

Make sure elasticsearch 6 is supported in phabricator
Open, NormalPublic

Description

Im filling this task to make sure elasticsearch gets supported with phabricator.

At one point wmf may upgrade to this version but i doint see this happening any time soon.

But best we support it :).

Event Timeline

Paladox created this task.Nov 27 2017, 12:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 27 2017, 12:36 PM
Aklapper triaged this task as Lowest priority.Nov 27 2017, 1:12 PM
Aklapper moved this task from To Triage to Misc on the Phabricator board.

It doesn't look like there are any changes that affect Phabricator. See Breaking Changes in ES 6.0 and Search Changes in ES 6.0

Thanks for looking into this. I can test this update on one of the test instances :).

Tested locally and seems to work except when generating the index

root@phabricator:/srv/phab/phabricator# bin/search init
Initializing search service "Elasticsearch".
Service index does not exist, creating...
[2017-11-27 17:40:39] EXCEPTION: (HTTPFutureHTTPResponseStatus) [HTTP/400] 
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [TASK]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields ... at [<phutil>/src/future/http/BaseHTTPFuture.php:351]
arcanist(), phabricator(), phutil(), security(), sprint(), wmf-ext-misc()
  #0 BaseHTTPFuture::parseRawHTTPResponse(string) called at [<phutil>/src/future/http/HTTPSFuture.php:418]
  #1 HTTPSFuture::isReady() called at [<phutil>/src/future/Future.php:37]
  #2 Future::resolve() called at [<phutil>/src/future/http/BaseHTTPFuture.php:291]
  #3 BaseHTTPFuture::resolvex() called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:581]
  #4 PhabricatorElasticFulltextStorageEngine::executeRequest(PhabricatorElasticsearchHost, string, array, string) called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:539]
  #5 PhabricatorElasticFulltextStorageEngine::initIndex() called at [<phabricator>/src/applications/search/management/PhabricatorSearchManagementInitWorkflow.php:40]
  #6 PhabricatorSearchManagementInitWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441]
  #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333]
  #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/search/manage_search.php:21]

I now get

[2017-11-27 19:42:29] EXCEPTION: (HTTPFutureHTTPResponseStatus) [HTTP/400] 
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [phabricator] as the final mapping would have more than 1 type: [TASK, FITV, POST, CDTL, CMIT, USER, MOCK, ATOM, QUES, REPO, BOOK, OPKG, WIKI, PROJ, BLOG, DREV, CEVT]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [phabricator] as the final mapping would have more than 1 type: [TASK, FITV, POST, CDTL, CMIT, USER, MOCK, ATOM, QUES, REPO, BOOK, OPKG, WIKI, PROJ, BLOG, DREV, CEVT]"},... at [<phutil>/src/future/http/BaseHTTPFuture.php:351]
arcanist(), phabricator(), phutil(), security(), sprint(), wmf-ext-misc()
  #0 BaseHTTPFuture::parseRawHTTPResponse(string) called at [<phutil>/src/future/http/HTTPSFuture.php:418]
  #1 HTTPSFuture::isReady() called at [<phutil>/src/future/Future.php:37]
  #2 Future::resolve() called at [<phutil>/src/future/http/BaseHTTPFuture.php:291]
  #3 BaseHTTPFuture::resolvex() called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:591]
  #4 PhabricatorElasticFulltextStorageEngine::executeRequest(PhabricatorElasticsearchHost, string, array, string) called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:549]
  #5 PhabricatorElasticFulltextStorageEngine::initIndex() called at [<phabricator>/src/applications/search/management/PhabricatorSearchManagementInitWorkflow.php:40]
  #6 PhabricatorSearchManagementInitWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441]
  #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333]
  #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/search/manage_search.php:21]

Yes elasticsearch 6 made some fairly drastic changes from the user perspective, but your use case is limited enough it shouldn't be too hard to handle. Basically you will need to unify the mapping into a single type, add a keyword field to indicate the index type, and add appropriate filters to the queries on that field. Under the hood this is what elasticsearch was already doing, they just decided to push this out of elasticsearch and onto applications.

@EBernhardson Any idea how soon we will need to care about ES6.x changes? I'll prioritize this if it's on the radar but otherwise I think I'll leave it alone until an upgrade is in the foreseeable future.

6.x is a ways out. They've made some fairly large changes that will require a bunch of work on our end to upgrade, so probably at least 6 months.

mmodell claimed this task.Jan 31 2018, 11:06 PM
mmodell removed mmodell as the assignee of this task.Apr 23 2018, 4:30 PM

6 months is approaching :) Still don't have a date, but am doing some preliminary work resolving current problems that would block the upgrade, specifically phabricator index in elasticsearch is created with include_in_all: false set on a variety of fields. The false is perfect, and the default value, but the property is to be removed in elasticsearch 6 and needs to be removed from the mapping.

The larger issue you are going to run into is that phabricator looks to utilize elasticsearch's multi "types", basically multiple schemas, in a single index. In elasticsearch 6 it will not be possible to create indices with more than one type. The most likely way forward for phabricator will be to add your own keyword field that indicates the type (user/opkg/etc), you will need to manually unify all the different type schemas into a single type, and you will need to adjust queries to filter/use that field wherever appropriate.

@EBernhardson thanks for the update and reminder. That doesn't sound too terribly difficult to deal with although it'll be a bit of work.

greg raised the priority of this task from Lowest to Normal.Jul 5 2018, 6:27 PM
greg added a subscriber: greg.

Re-prioritizing.

@EBernhardson still no date yet, right? And from what you said in T192614#4249710 are we actually OK with current status quo or does the multi-type index block us from using ES 6?

I am hoping to get things switched over by the end of Q1, but we still have some prep work in the pipeline so it's not a specific Q1 goal.

For status quo, yes, everything that phabricator currently does at query/indexing time will continue to work after the upgrade to elasticsearch 6. What will happen though is in 6 you can no longer create indices with multiple types like phabricator uses, so the current index schema would essentially be locked in until the phabricator code is update. This is not an ideal state, creating the (quite small) possibility that the index needs to be recreated to solve some problem but cannot.

mmodell claimed this task.Jul 5 2018, 10:50 PM
mmodell raised the priority of this task from Normal to High.Aug 7 2018, 12:53 AM

High priority due to the situation described in T181393#4401319

Aklapper moved this task from Misc to Infrastructure on the Phabricator board.Aug 29 2018, 6:03 PM
mmodell moved this task from Backlog to Working on it on the User-MModell board.Sep 10 2018, 4:54 PM
Alroilim closed this task as Declined.Feb 2 2019, 7:14 PM
Alroilim lowered the priority of this task from High to Lowest.
Alroilim removed mmodell as the assignee of this task.
Alroilim set Due Date to Feb 1 2019, 9:00 PM.
Alroilim updated the task description. (Show Details)
Alroilim removed subscribers: greg, EBernhardson, Gehel and 3 others.
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptFeb 2 2019, 7:14 PM
Paladox reopened this task as Open.Feb 2 2019, 7:18 PM
Paladox assigned this task to mmodell.
Paladox raised the priority of this task from Lowest to High.
Paladox removed Due Date.
Paladox updated the task description. (Show Details)
Paladox added subscribers: EBernhardson, greg, Gehel and 2 others.
Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptFeb 2 2019, 7:18 PM
EBernhardson added a comment.EditedFeb 4 2019, 4:47 PM

This upgrade should be happening late february or early march.

This upgrade is scheduled for next week, assuming this weeks train rolls forward as expected.

es6 patch was rolled back because at the time I think we were still on es5 and my code wasn't as forward/backward compatible as I thought it was. Still need to re-apply the patch for fully es6 index.

mmodell lowered the priority of this task from High to Normal.Wed, Jun 12, 9:44 PM