Page MenuHomePhabricator

Upgrade ElasticSearch to 1.7.5
Closed, ResolvedPublic

Description

1.7.5 is out. We're on an older 1.7.1 point release, so we should look at upgrading

Deb: https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.5.deb

Event Timeline

Reedy created this task.Dec 31 2015, 1:57 PM
Reedy raised the priority of this task from to Needs Triage.
Reedy updated the task description. (Show Details)
Reedy added a project: Elasticsearch.
Reedy added a subscriber: Reedy.
Restricted Application added a project: Discovery. · View Herald TranscriptDec 31 2015, 1:57 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
Reedy updated the task description. (Show Details)Dec 31 2015, 2:01 PM
Reedy set Security to None.
Reedy added a comment.Dec 31 2015, 4:53 PM

I presume, we can get ops to import the deb, and then start upgrading logstash etc?

Also blocks further updates to the elastica library

Hmm, the latest release of Elastica (which can be used on the wmf cluster[1]) seems to support 1.7.3 only (I don't know, which features don't work):

This release is compatible with elasticsearch 1.7.3.

(source: https://github.com/ruflin/Elastica/releases/tag/2.3.1)

[1] Any newer version of Elastica has a minimum dependency to php 5.4 (support for 5.3 was dropped in the 3.0.0-beta1 release, one more reason to drop php 5.3 support for MW, too *hust* :D).

Reedy added a comment.EditedJan 15 2016, 11:21 PM

We're on 2.2.0 atm

https://gerrit.wikimedia.org/r/#/c/261768/

https://github.com/ruflin/Elastica/releases/tag/2.2.1

2.2.1 supports PHP >= 5.3.3
https://github.com/ruflin/Elastica/blob/2.2.1/composer.json

So does 2.3.1, but it needs ES 1.7.2
https://github.com/ruflin/Elastica/blob/2.3.1/composer.json

3.0.1 needs ES 2.1.1...

Are the Elastica releases tied to a specific version of ES?

Reedy added a comment.Jan 17 2016, 8:59 PM

https://github.com/ruflin/Elastica/issues/1036#issuecomment-172360242

All 1.7 versions and even previous versions should not be a problem with 2.3.*. The version that is stated as compatible is the one that was tested. In general as long as there are no major changes on the elasticsearch side, Elastica releases are compatible with previous and later release. Best for 1.7.4 is definitively 2.3.1

EBernhardson moved this task from Needs triage to Search on the Discovery board.Feb 11 2016, 11:24 PM
Gehel added a subscriber: Gehel.Feb 15 2016, 2:21 PM
Gehel added a subscriber: dcausse.Feb 15 2016, 2:24 PM

To do (as discussed with @dcausse):

  1. upgrade Vagrant so that we can test in dev, with no impact
  2. upgrade Cindy (browser test bot) to validate
  3. upgrade labs
  4. upgrade prod

Let me know if I forgot something...

dcausse renamed this task from Upgrade ElasticSearch to 1.7.4 to Upgrade ElasticSearch to 1.7.5.Feb 15 2016, 2:25 PM
dcausse updated the task description. (Show Details)
Gehel added a comment.Feb 16 2016, 8:46 AM

After talking with Ops, seems that we want to do some testing before uploading new elasticsearch version to our apt repo. List of tasks looks like:

  1. upgrade Vagrant so that we can test in dev, with no impact
  2. upgrade Cindy (browser test bot) to validate
  3. upgrade labs
  4. upload new elasticsearch version to reprepro
  5. upgrade prod

Which means that I'm not sure of how we want to upgrade Vagrant. We could wget the .deb instead of getting it from reprepro, but that looks fairly ugly to me.

Which means that I'm not sure of how we want to upgrade Vagrant. We could wget the .deb instead of getting it from reprepro, but that looks fairly ugly to me.

We've been known to do this in production so we can prevent any unintended upgrades and do one machine at a time.

You may want to upgrade the production logstash boxes before doing the elasticsearch cluster itself too, but that's upto you :)

Gehel added a comment.Feb 17 2016, 5:59 PM

Testing in mediawiki-vagrant locally, with elasticsearch 1.7.1, I have the following tests in error:

Failing Scenarios:
cucumber features/create_new_page.feature:7 # Scenario: Something something
cucumber features/create_new_page.feature:23 # Scenario: boolean operators in bad positions in the query are ignored so you get the option to create a new page
cucumber features/create_new_page.feature:61 # Scenario: boolean operators in bad positions in the query are ignored but if there are other valid operators then you don't get the option to create a new page
cucumber features/create_new_page.feature:96 # Scenario: Wildcards can't start a term but they aren't valid titles so you still don't get the link to create an article
cucumber features/create_new_page.feature:96 # Scenario: Wildcards can't start a term but they aren't valid titles so you still don't get the link to create an article
cucumber features/frozen_index_api.feature:30 # Scenario: Updates to OtherIndex are delayed
cucumber features/prefix_search_api.feature:3 # Scenario: Suggestions don't appear when you search for a string that is too long
cucumber features/update_general_api.feature:21 # Scenario: Pages containing altered template are updated in the index

after upgrade of elasticsearch to 1.7.5 I have the following tests in error:

Failing Scenarios:
cucumber features/create_new_page.feature:7 # Scenario: Something something
cucumber features/create_new_page.feature:23 # Scenario: boolean operators in bad positions in the query are ignored so you get the option to create a new page
cucumber features/create_new_page.feature:61 # Scenario: boolean operators in bad positions in the query are ignored but if there are other valid operators then you don't get the option to create a new page
cucumber features/create_new_page.feature:96 # Scenario: Wildcards can't start a term but they aren't valid titles so you still don't get the link to create an article
cucumber features/create_new_page.feature:96 # Scenario: Wildcards can't start a term but they aren't valid titles so you still don't get the link to create an article
cucumber features/prefix_search_api.feature:3 # Scenario: Suggestions don't appear when you search for a string that is too long

So all tests failing after upgrade were already in error before upgrade. The following tests have been automagically fixed (probably completely unrelated to the upgrade):

cucumber features/frozen_index_api.feature:30 # Scenario: Updates to OtherIndex are delayed
cucumber features/update_general_api.feature:21 # Scenario: Pages containing altered template are updated in the index
Gehel added a subscriber: bd808.Feb 23 2016, 9:51 AM

v 1.7.5 has been applied on Discovery's elasticsearch cluster in labs.

With the help of @bd808 the version 1.7.5 has been also applied on the logstash ES in labs. Upgrading the logstash instances seems the right thing to do.

We can bundle tomorrow's logstash* kernel upgrade to also make the upgrade, logstash* is currently at 1.7.1 and this is supposed to be a compatible 1.7.x series and since tests in labs went fine.

Reedy added a comment.Feb 23 2016, 9:57 AM

We can bundle tomorrow's logstash* kernel upgrade to also make the upgrade, logstash* is currently at 1.7.1 and this is supposed to be a compatible 1.7.x series and since tests in labs went fine.

Sounds good. I presume they'll be done at one at a time, rebooted, wait for the cluster to recover/resync and move onto the next box?

Sounds good. I presume they'll be done at one at a time, rebooted, wait for the cluster to recover/resync and move onto the next box?

Yeah, nodes will be rebooted one at a time and we'll proceed only when the cluster has recovered to "green".

Starting tomorrow at 20:00 UTC

Change 272721 had a related patch set uploaded (by Gehel):
Upgrade elastic search to 1.7.5

https://gerrit.wikimedia.org/r/272721

Change 272721 merged by Gehel:
Upgrade elastic search to 1.7.5

https://gerrit.wikimedia.org/r/272721

Mentioned in SAL [2016-02-29T12:16:43Z] <gehel> elastic2001.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-02-29T14:16:40Z] <gehel> elastic2001.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-02-29T18:01:34Z] <gehel> elastic2002.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-02-29T19:22:20Z] <gehel> elastic2003.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-02-29T20:21:34Z] <gehel> elastic2004.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T09:46:29Z] <gehel> elastic2016.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T10:42:38Z] <gehel> elastic2017.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T11:40:13Z] <gehel> elastic2018.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T12:37:30Z] <gehel> elastic2019.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T13:53:43Z] <gehel> elastic2020.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T14:34:03Z] <gehel> elastic2021.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T15:45:35Z] <gehel> elastic2022.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T16:40:43Z] <gehel> elastic2023.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T17:52:14Z] <gehel> elastic2024.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T21:29:36Z] <gehel> elastic1001.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T08:48:30Z] <gehel> elastic1003.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T10:16:00Z] <gehel> elastic1004.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T13:23:33Z] <gehel> elastic1005.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T14:32:25Z] <gehel> elastic1006.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T15:15:34Z] <gehel> elastic1007.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T16:34:08Z] <gehel> elastic1008.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T17:20:58Z] <gehel> elastic1009.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T18:58:46Z] <gehel> elastic1010.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T20:35:46Z] <gehel> elastic1011.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T10:07:48Z] <gehel> elastic1022.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T12:02:06Z] <gehel> elastic1023.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T12:48:56Z] <gehel> elastic1024.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T13:47:18Z] <gehel> elastic1025.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T14:34:38Z] <gehel> elastic1026.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T15:42:43Z] <gehel> elastic1027.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T16:25:50Z] <gehel> elastic1028.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T17:09:55Z] <gehel> elastic1029.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T18:41:56Z] <gehel> elastic1030.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T19:44:23Z] <gehel> elastic1031.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

estest100{1..4}.eqiad.wmflabs have also been upgraded. Nobelium has not (as suggested by @EBernhardson)

Deskana closed this task as Resolved.Mar 5 2016, 1:46 AM
Deskana added a subscriber: Deskana.

Yay!

Restricted Application added a project: Discovery-Search. · View Herald TranscriptApr 5 2016, 10:53 PM