@bd808 since I'm upgrading the cirrus clusters, do you want me to take care of the labs one as well?

In T164842#3298317, @Gehel wrote:

@bd808 since I'm upgrading the cirrus clusters, do you want me to take care of the labs one as well?

Thanks for the offer @Gehel. This cluster in Tool Labs is pretty small and Stashbot and the SAL and bash tools I maintain are the main users of it, so I don't mind being the one to do the upgrade. I'll be in a better position to test things than most people would. I may ask for your help if I run into difficulty in the updates though.

Mentioned in SAL (#wikimedia-cloud) [2017-06-22T16:19:56Z] <bd808> Backed up elasticsearch indexes to personal laptop using elasticdump incase T164842 goes horribly wrong

Mentioned in SAL (#wikimedia-cloud) [2017-06-22T17:03:07Z] <bd808> Rolled back attempt at Elasticsearch upgrade. Indices need to be rebuilt with 2.x before 5.x can be installed. T164842

Aborted upgrade because of this error:

Caused by: java.lang.IllegalStateException: The index [[bash/8j-7PJe8TFWSfT5pbViVWg]] was created with version [1.7.1] but the minimum compatible version is [2.0.0-beta1]. It should be re-indexed in Elasticsearch 2.x before upgrading to 5.3.2.

I need to install the migration plugin and find out all the things that need to be fixed before trying again. Once things are ready, the rough steps needed are:

• sudo -i puppet agent --disable 'upgrading elasticsearch'
• sudo service elasticsearch stop
• Set elasticsearch::version: 5 in hiera
• sudo apt-get install elasticsearch openjdk-8-jdk
• sudo apt-get purge openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless
• sudo -i puppet agent --enable
• sudo -i puppet agent --test --verbose
• sudo service elasticsearch status and/or sudo journalctl -l --no-pager -u elasticsearch.service

Reindexing has been done for the indexes that were created with ElasticSearch <2.x. Migration helper is showing a few issues, but I think they are things that will auto-update or be handled by the Puppet changes that setting elasticsearch::version: 5 does.

Tentatively planning to upgrade the cluster on 2017-06-29 around 21:00Z following the MediaWiki train deploy.

Mentioned in SAL (#wikimedia-cloud) [2017-07-13T21:25:29Z] <bd808> Upgrading ElasticSearch cluster for T164842. There will be service interruptions

Change 365178 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/tools/stashbot@master] Update for ElasticSearch 5.x

https://gerrit.wikimedia.org/r/365178

Change 365178 merged by jenkins-bot:
[labs/tools/stashbot@master] Update for ElasticSearch 5.x

https://gerrit.wikimedia.org/r/365178

Mentioned in SAL (#wikimedia-cloud) [2017-07-13T22:02:33Z] <bd808> Updated to 3c3810c for T164842