Install ES 5.x to relforge100[12]
Open, NormalPublic

Description

Testing elastic5 in beta cluster is not easy, we could use relforge to do early evaluations of cirrus+es5 on large indices.

Gehel claimed this task.
Gehel added a subscriber: dcausse.Jan 25 2017, 3:25 PM

After discussion with @dcausse, the following steps are needed to install ES5 on relforge:

  • deploy gerrit change 333969
  • remove indices using jieba plugin (curl localhost:9200/_settings | jq . | grep jieba)
  • remove jieba, ltr and swift plugins and restart elasticsearch (on 2.x version), check that all indices still load correctly
  • upgrade remaining plugins manually and upgrade elasticsearch (require a full cluster restart).

Mentioned in SAL (#wikimedia-operations) [2017-01-25T15:27:17Z] <gehel> deleting indices using jieba plugin from relforge - T156150

Mentioned in SAL (#wikimedia-operations) [2017-01-25T15:28:38Z] <gehel> removing jieba / ltr / swift plugins from elasticsearch relforge - T156150

dcausse added subscribers: debt, Jdrewniak.EditedJan 25 2017, 3:29 PM

I forgot to mention this morning that upgrading to elastic5 will break sistersearch (the interwiki demo @Jdrewniak is working on).
It's unclear to me what are the priorities here.
I think the cirrussearch es5 branch is mostly ready to support simple searches so we should be able to migrate sistersearch the es5 branch.
I just wanted to point out that it may cause some disruption in the work Jan is doing.
@debt, @Jdrewniak thoughts?

dcausse edited the task description. (Show Details)Jan 25 2017, 3:33 PM

We want to share the sistersearch demo (http://sistersearch.wmflabs.org/) on a few mailing lists to get feedback, so I wouldn't break it just yet. Once we run the A/B test on a few production wikis we won't need the labs instance, so then it should be fine to break then :)

Planning to upgrade the relforge servers beginning of next week. Expecting a short downtime for sistersearch, after which David or I will pull the es5 branches of CirrusSearch/Elastica to sistersearch to get things going again.

Change 339423 had a related patch set uploaded (by Gehel):
relforge: upgrade elasticsearch to v5.2.1

https://gerrit.wikimedia.org/r/339423

Mentioned in SAL (#wikimedia-operations) [2017-02-23T15:45:10Z] <gehel> banning relforge1001 from clsuter to prepare for ES5 upgrade - T156150

Mentioned in SAL (#wikimedia-operations) [2017-02-23T16:19:12Z] <gehel> unban relforge1001 - T156150

Mentioned in SAL (#wikimedia-operations) [2017-02-23T16:19:43Z] <gehel> starting upgrade relforge cluster to elasticsearch 5.2.1 - expect significant downtime - T156150

Change 339423 merged by Gehel:
relforge: upgrade elasticsearch to v5.2.1

https://gerrit.wikimedia.org/r/339423

Mentioned in SAL (#wikimedia-operations) [2017-02-23T17:37:26Z] <gehel> removing old prod indices from relforge1002 (jawikiprod_content, enprodwiki_content, ruwikiprod_content) - T156150

Mentioned in SAL (#wikimedia-operations) [2017-02-23T17:40:34Z] <gehel> removing old prod indices from relforge1002 - T156150

Gehel added a comment.Feb 23 2017, 5:45 PM

deletion of old "prod" indices created with ES 1.3.4 was required.

Mentioned in SAL (#wikimedia-operations) [2017-03-06T10:36:02Z] <gehel> upgrade to elasticsearch 5.2.2 on relforge cluster - T156150

Mentioned in SAL (#wikimedia-operations) [2017-03-13T09:23:03Z] <gehel> downgrading elasticsearch to v5.1.2 on relforge, a full reindex will be needed - T156150