I presume, we can get ops to import the deb, and then start upgrading logstash etc?

Also blocks further updates to the elastica library

Hmm, the latest release of Elastica (which can be used on the wmf cluster[1]) seems to support 1.7.3 only (I don't know, which features don't work):

This release is compatible with elasticsearch 1.7.3.

(source: https://github.com/ruflin/Elastica/releases/tag/2.3.1)

[1] Any newer version of Elastica has a minimum dependency to php 5.4 (support for 5.3 was dropped in the 3.0.0-beta1 release, one more reason to drop php 5.3 support for MW, too *hust* :D).

We're on 2.2.0 atm

https://gerrit.wikimedia.org/r/#/c/261768/

https://github.com/ruflin/Elastica/releases/tag/2.2.1

2.2.1 supports PHP >= 5.3.3
https://github.com/ruflin/Elastica/blob/2.2.1/composer.json

So does 2.3.1, but it needs ES 1.7.2
https://github.com/ruflin/Elastica/blob/2.3.1/composer.json

3.0.1 needs ES 2.1.1...

Are the Elastica releases tied to a specific version of ES?

https://github.com/ruflin/Elastica/issues/1036#issuecomment-172360242

All 1.7 versions and even previous versions should not be a problem with 2.3.*. The version that is stated as compatible is the one that was tested. In general as long as there are no major changes on the elasticsearch side, Elastica releases are compatible with previous and later release. Best for 1.7.4 is definitively 2.3.1

To do (as discussed with @dcausse):

1. upgrade Vagrant so that we can test in dev, with no impact
2. upgrade Cindy (browser test bot) to validate
3. upgrade labs
4. upgrade prod

Let me know if I forgot something...

After talking with Ops, seems that we want to do some testing before uploading new elasticsearch version to our apt repo. List of tasks looks like:

1. upgrade Vagrant so that we can test in dev, with no impact
2. upgrade Cindy (browser test bot) to validate
3. upgrade labs
4. upload new elasticsearch version to reprepro
5. upgrade prod

Which means that I'm not sure of how we want to upgrade Vagrant. We could wget the .deb instead of getting it from reprepro, but that looks fairly ugly to me.

In T122697#2030447, @Gehel wrote:

Which means that I'm not sure of how we want to upgrade Vagrant. We could wget the .deb instead of getting it from reprepro, but that looks fairly ugly to me.

We've been known to do this in production so we can prevent any unintended upgrades and do one machine at a time.

You may want to upgrade the production logstash boxes before doing the elasticsearch cluster itself too, but that's upto you :)

Testing in mediawiki-vagrant locally, with elasticsearch 1.7.1, I have the following tests in error:

Failing Scenarios:
cucumber features/create_new_page.feature:7 # Scenario: Something something
cucumber features/create_new_page.feature:23 # Scenario: boolean operators in bad positions in the query are ignored so you get the option to create a new page
cucumber features/create_new_page.feature:61 # Scenario: boolean operators in bad positions in the query are ignored but if there are other valid operators then you don't get the option to create a new page
cucumber features/create_new_page.feature:96 # Scenario: Wildcards can't start a term but they aren't valid titles so you still don't get the link to create an article
cucumber features/create_new_page.feature:96 # Scenario: Wildcards can't start a term but they aren't valid titles so you still don't get the link to create an article
cucumber features/frozen_index_api.feature:30 # Scenario: Updates to OtherIndex are delayed
cucumber features/prefix_search_api.feature:3 # Scenario: Suggestions don't appear when you search for a string that is too long
cucumber features/update_general_api.feature:21 # Scenario: Pages containing altered template are updated in the index

after upgrade of elasticsearch to 1.7.5 I have the following tests in error:

Failing Scenarios:
cucumber features/create_new_page.feature:7 # Scenario: Something something
cucumber features/create_new_page.feature:23 # Scenario: boolean operators in bad positions in the query are ignored so you get the option to create a new page
cucumber features/create_new_page.feature:61 # Scenario: boolean operators in bad positions in the query are ignored but if there are other valid operators then you don't get the option to create a new page
cucumber features/create_new_page.feature:96 # Scenario: Wildcards can't start a term but they aren't valid titles so you still don't get the link to create an article
cucumber features/create_new_page.feature:96 # Scenario: Wildcards can't start a term but they aren't valid titles so you still don't get the link to create an article
cucumber features/prefix_search_api.feature:3 # Scenario: Suggestions don't appear when you search for a string that is too long

So all tests failing after upgrade were already in error before upgrade. The following tests have been automagically fixed (probably completely unrelated to the upgrade):

cucumber features/frozen_index_api.feature:30 # Scenario: Updates to OtherIndex are delayed
cucumber features/update_general_api.feature:21 # Scenario: Pages containing altered template are updated in the index

v 1.7.5 has been applied on Discovery's elasticsearch cluster in labs.

With the help of @bd808 the version 1.7.5 has been also applied on the logstash ES in labs. Upgrading the logstash instances seems the right thing to do.

We can bundle tomorrow's logstash* kernel upgrade to also make the upgrade, logstash* is currently at 1.7.1 and this is supposed to be a compatible 1.7.x series and since tests in labs went fine.

In T122697#2054635, @MoritzMuehlenhoff wrote:

We can bundle tomorrow's logstash* kernel upgrade to also make the upgrade, logstash* is currently at 1.7.1 and this is supposed to be a compatible 1.7.x series and since tests in labs went fine.

Sounds good. I presume they'll be done at one at a time, rebooted, wait for the cluster to recover/resync and move onto the next box?

In T122697#2054645, @Reedy wrote:

Sounds good. I presume they'll be done at one at a time, rebooted, wait for the cluster to recover/resync and move onto the next box?

Yeah, nodes will be rebooted one at a time and we'll proceed only when the cluster has recovered to "green".

Starting tomorrow at 20:00 UTC

Change 272721 had a related patch set uploaded (by Gehel):
Upgrade elastic search to 1.7.5

https://gerrit.wikimedia.org/r/272721

Change 272721 merged by Gehel:
Upgrade elastic search to 1.7.5

https://gerrit.wikimedia.org/r/272721

Mentioned in SAL [2016-02-29T12:16:43Z] <gehel> elastic2001.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-02-29T14:16:40Z] <gehel> elastic2001.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-02-29T18:01:34Z] <gehel> elastic2002.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-02-29T19:22:20Z] <gehel> elastic2003.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-02-29T20:21:34Z] <gehel> elastic2004.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T09:46:29Z] <gehel> elastic2016.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T10:42:38Z] <gehel> elastic2017.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T11:40:13Z] <gehel> elastic2018.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T12:37:30Z] <gehel> elastic2019.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T13:53:43Z] <gehel> elastic2020.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T14:34:03Z] <gehel> elastic2021.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T15:45:35Z] <gehel> elastic2022.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T16:40:43Z] <gehel> elastic2023.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T17:52:14Z] <gehel> elastic2024.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-01T21:29:36Z] <gehel> elastic1001.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T08:48:30Z] <gehel> elastic1003.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T10:16:00Z] <gehel> elastic1004.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T13:23:33Z] <gehel> elastic1005.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T14:32:25Z] <gehel> elastic1006.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T15:15:34Z] <gehel> elastic1007.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T16:34:08Z] <gehel> elastic1008.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T17:20:58Z] <gehel> elastic1009.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T18:58:46Z] <gehel> elastic1010.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-02T20:35:46Z] <gehel> elastic1011.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T10:07:48Z] <gehel> elastic1022.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T12:02:06Z] <gehel> elastic1023.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T12:48:56Z] <gehel> elastic1024.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T13:47:18Z] <gehel> elastic1025.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T14:34:38Z] <gehel> elastic1026.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T15:42:43Z] <gehel> elastic1027.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T16:25:50Z] <gehel> elastic1028.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T17:09:55Z] <gehel> elastic1029.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T18:41:56Z] <gehel> elastic1030.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

Mentioned in SAL [2016-03-03T19:44:23Z] <gehel> elastic1031.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)

estest100{1..4}.eqiad.wmflabs have also been upgraded. Nobelium has not (as suggested by @EBernhardson)