Upgrade Tool Labs elasticsearch to 5.x
Closed, ResolvedPublic

Description

Follow upgrade path of production Elasticsearch clusters by upgrading to the 5.x version in Wikimedia's apt repo.

bd808 created this task.May 9 2017, 3:04 PM
Restricted Application added projects: Cloud-Services, User-bd808. · View Herald TranscriptMay 9 2017, 3:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel added a subscriber: Gehel.May 29 2017, 1:51 PM

@bd808 since I'm upgrading the cirrus clusters, do you want me to take care of the labs one as well?

@bd808 since I'm upgrading the cirrus clusters, do you want me to take care of the labs one as well?

Thanks for the offer @Gehel. This cluster in Tool Labs is pretty small and Stashbot and the SAL and bash tools I maintain are the main users of it, so I don't mind being the one to do the upgrade. I'll be in a better position to test things than most people would. I may ask for your help if I run into difficulty in the updates though.

Mentioned in SAL (#wikimedia-cloud) [2017-06-22T16:19:56Z] <bd808> Backed up elasticsearch indexes to personal laptop using elasticdump incase T164842 goes horribly wrong

Mentioned in SAL (#wikimedia-cloud) [2017-06-22T17:03:07Z] <bd808> Rolled back attempt at Elasticsearch upgrade. Indices need to be rebuilt with 2.x before 5.x can be installed. T164842

bd808 added a comment.Jun 22 2017, 5:13 PM

Aborted upgrade because of this error:

Caused by: java.lang.IllegalStateException: The index [[bash/8j-7PJe8TFWSfT5pbViVWg]] was created with version [1.7.1] but the minimum compatible version is [2.0.0-beta1]. It should be re-indexed in Elasticsearch 2.x before upgrading to 5.3.2.

I need to install the migration plugin and find out all the things that need to be fixed before trying again. Once things are ready, the rough steps needed are:

  • sudo -i puppet agent --disable 'upgrading elasticsearch'
  • sudo service elasticsearch stop
  • Set elasticsearch::version: 5 in hiera
  • sudo apt-get install elasticsearch openjdk-8-jdk
  • sudo apt-get purge openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless
  • sudo -i puppet agent --enable
  • sudo -i puppet agent --test --verbose
  • sudo service elasticsearch status and/or sudo journalctl -l --no-pager -u elasticsearch.service
bd808 added a comment.Mon, Jun 26, 4:42 PM

Reindexing has been done for the indexes that were created with ElasticSearch <2.x. Migration helper is showing a few issues, but I think they are things that will auto-update or be handled by the Puppet changes that setting elasticsearch::version: 5 does.

Tentatively planning to upgrade the cluster on 2017-06-29 around 21:00Z following the MediaWiki train deploy.

Mentioned in SAL (#wikimedia-cloud) [2017-07-13T21:25:29Z] <bd808> Upgrading ElasticSearch cluster for T164842. There will be service interruptions

Change 365178 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/tools/stashbot@master] Update for ElasticSearch 5.x

https://gerrit.wikimedia.org/r/365178

Change 365178 merged by jenkins-bot:
[labs/tools/stashbot@master] Update for ElasticSearch 5.x

https://gerrit.wikimedia.org/r/365178

Mentioned in SAL (#wikimedia-cloud) [2017-07-13T22:02:33Z] <bd808> Updated to 3c3810c for T164842

bd808 closed this task as Resolved.Fri, Jul 14, 12:39 AM