Im filling this task to make sure elasticsearch gets supported with phabricator.
At one point wmf may upgrade to this version but i doint see this happening any time soon.
But best we support it :).
Description
Description
Revisions and Commits
Revisions and Commits
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | EBernhardson | T183281 [epic] ELK upgrade to 6.x (elasticsearch, kibana, logstash) | |||
| Resolved | None | T183282 [epic] Search cluster upgrade to 6.x | |||
| Resolved | None | T194199 [Epic] Prepare for Elasticsearch 6 upgrade | |||
| Declined | mmodell | T181393 Make sure elasticsearch 6 is supported in phabricator | |||
| Resolved | mmodell | T230787 Make search context highlights work with the ferret search engine |
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions
It doesn't look like there are any changes that affect Phabricator. See Breaking Changes in ES 6.0 and Search Changes in ES 6.0
Comment Actions
Thanks for looking into this. I can test this update on one of the test instances :).
Comment Actions
Tested locally and seems to work except when generating the index
root@phabricator:/srv/phab/phabricator# bin/search init
Initializing search service "Elasticsearch".
Service index does not exist, creating...
[2017-11-27 17:40:39] EXCEPTION: (HTTPFutureHTTPResponseStatus) [HTTP/400]
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [TASK]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields ... at [<phutil>/src/future/http/BaseHTTPFuture.php:351]
arcanist(), phabricator(), phutil(), security(), sprint(), wmf-ext-misc()
#0 BaseHTTPFuture::parseRawHTTPResponse(string) called at [<phutil>/src/future/http/HTTPSFuture.php:418]
#1 HTTPSFuture::isReady() called at [<phutil>/src/future/Future.php:37]
#2 Future::resolve() called at [<phutil>/src/future/http/BaseHTTPFuture.php:291]
#3 BaseHTTPFuture::resolvex() called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:581]
#4 PhabricatorElasticFulltextStorageEngine::executeRequest(PhabricatorElasticsearchHost, string, array, string) called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:539]
#5 PhabricatorElasticFulltextStorageEngine::initIndex() called at [<phabricator>/src/applications/search/management/PhabricatorSearchManagementInitWorkflow.php:40]
#6 PhabricatorSearchManagementInitWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441]
#7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333]
#8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/search/manage_search.php:21]Comment Actions
I now get
[2017-11-27 19:42:29] EXCEPTION: (HTTPFutureHTTPResponseStatus) [HTTP/400]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [phabricator] as the final mapping would have more than 1 type: [TASK, FITV, POST, CDTL, CMIT, USER, MOCK, ATOM, QUES, REPO, BOOK, OPKG, WIKI, PROJ, BLOG, DREV, CEVT]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [phabricator] as the final mapping would have more than 1 type: [TASK, FITV, POST, CDTL, CMIT, USER, MOCK, ATOM, QUES, REPO, BOOK, OPKG, WIKI, PROJ, BLOG, DREV, CEVT]"},... at [<phutil>/src/future/http/BaseHTTPFuture.php:351]
arcanist(), phabricator(), phutil(), security(), sprint(), wmf-ext-misc()
#0 BaseHTTPFuture::parseRawHTTPResponse(string) called at [<phutil>/src/future/http/HTTPSFuture.php:418]
#1 HTTPSFuture::isReady() called at [<phutil>/src/future/Future.php:37]
#2 Future::resolve() called at [<phutil>/src/future/http/BaseHTTPFuture.php:291]
#3 BaseHTTPFuture::resolvex() called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:591]
#4 PhabricatorElasticFulltextStorageEngine::executeRequest(PhabricatorElasticsearchHost, string, array, string) called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:549]
#5 PhabricatorElasticFulltextStorageEngine::initIndex() called at [<phabricator>/src/applications/search/management/PhabricatorSearchManagementInitWorkflow.php:40]
#6 PhabricatorSearchManagementInitWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441]
#7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333]
#8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/search/manage_search.php:21]Comment Actions
@mmodell It seems that ^^ is a breaking change in 6.x https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html
Comment Actions
Yes elasticsearch 6 made some fairly drastic changes from the user perspective, but your use case is limited enough it shouldn't be too hard to handle. Basically you will need to unify the mapping into a single type, add a keyword field to indicate the index type, and add appropriate filters to the queries on that field. Under the hood this is what elasticsearch was already doing, they just decided to push this out of elasticsearch and onto applications.
Comment Actions
@EBernhardson Any idea how soon we will need to care about ES6.x changes? I'll prioritize this if it's on the radar but otherwise I think I'll leave it alone until an upgrade is in the foreseeable future.
Comment Actions
6.x is a ways out. They've made some fairly large changes that will require a bunch of work on our end to upgrade, so probably at least 6 months.
Comment Actions
6 months is approaching :) Still don't have a date, but am doing some preliminary work resolving current problems that would block the upgrade, specifically phabricator index in elasticsearch is created with include_in_all: false set on a variety of fields. The false is perfect, and the default value, but the property is to be removed in elasticsearch 6 and needs to be removed from the mapping.
The larger issue you are going to run into is that phabricator looks to utilize elasticsearch's multi "types", basically multiple schemas, in a single index. In elasticsearch 6 it will not be possible to create indices with more than one type. The most likely way forward for phabricator will be to add your own keyword field that indicates the type (user/opkg/etc), you will need to manually unify all the different type schemas into a single type, and you will need to adjust queries to filter/use that field wherever appropriate.
Comment Actions
@EBernhardson thanks for the update and reminder. That doesn't sound too terribly difficult to deal with although it'll be a bit of work.
Comment Actions
Re-prioritizing.
@EBernhardson still no date yet, right? And from what you said in T192614#4249710 are we actually OK with current status quo or does the multi-type index block us from using ES 6?
Comment Actions
I am hoping to get things switched over by the end of Q1, but we still have some prep work in the pipeline so it's not a specific Q1 goal.
For status quo, yes, everything that phabricator currently does at query/indexing time will continue to work after the upgrade to elasticsearch 6. What will happen though is in 6 you can no longer create indices with multiple types like phabricator uses, so the current index schema would essentially be locked in until the phabricator code is update. This is not an ideal state, creating the (quite small) possibility that the index needs to be recreated to solve some problem but cannot.
Comment Actions
Removed include_in_all in commit rPHAB5577304cd98d: Remove 'include_in_all' for es6 compatibility.
Comment Actions
This upgrade is scheduled for next week, assuming this weeks train rolls forward as expected.
Comment Actions
es6 patch was rolled back because at the time I think we were still on es5 and my code wasn't as forward/backward compatible as I thought it was. Still need to re-apply the patch for fully es6 index.
Comment Actions
This won't be needed if we are successful with T230787: Make search context highlights work with the ferret search engine
Comment Actions
root@phabricator-prod-1001:/srv/phab/phabricator# bin/search init
Initializing search service "Elasticsearch".
Service index does not exist, creating...
[2020-05-04 13:00:26] EXCEPTION: (HTTPFutureHTTPResponseStatus) [HTTP/400]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [phabricator] as the final mapping would have more than 1 type: [TASK, FITV, POST, CDTL, DSHB, CMIT, USER, MOCK, ATOM, QUES, REPO, BOOK, DSHP, OPKG, WIKI, PROJ, BLOG, DREV, CEVT, PRTL]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [phabricator] as the final mapping would have more than 1 type: [TASK, FITV, POST, CDTL, DSHB, CMIT, USER, MOCK, ATOM, QUES, REPO, BOOK, DSHP, OPKG, WI... at [<phutil>/src/future/http/BaseHTTPFuture.php:351]
arcanist(), ava(), phabricator(), phutil(), security(), sprint(), translations(), wmf-ext-misc()
#0 BaseHTTPFuture::parseRawHTTPResponse(string) called at [<phutil>/src/future/http/HTTPSFuture.php:497]
#1 HTTPSFuture::isReady() called at [<phutil>/src/future/Future.php:37]
#2 Future::resolve() called at [<phutil>/src/future/http/BaseHTTPFuture.php:291]
#3 BaseHTTPFuture::resolvex() called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:542]
#4 PhabricatorElasticFulltextStorageEngine::executeRequest(PhabricatorElasticsearchHost, string, string, string) called at [<phabricator>/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:500]
#5 PhabricatorElasticFulltextStorageEngine::initIndex() called at [<phabricator>/src/applications/search/management/PhabricatorSearchManagementInitWorkflow.php:40]
#6 PhabricatorSearchManagementInitWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:457]
#7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:349]
#8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/search/manage_search.php:21]That's the error i get.