Page MenuHomePhabricator

Upgrade Tool Labs elasticsearch to 5.x
Closed, ResolvedPublic

Description

Follow upgrade path of production Elasticsearch clusters by upgrading to the 5.x version in Wikimedia's apt repo.

Event Timeline

bd808 created this task.May 9 2017, 3:04 PM
Restricted Application added projects: Cloud-Services, User-bd808. · View Herald TranscriptMay 9 2017, 3:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel added a subscriber: Gehel.May 29 2017, 1:51 PM

@bd808 since I'm upgrading the cirrus clusters, do you want me to take care of the labs one as well?

@bd808 since I'm upgrading the cirrus clusters, do you want me to take care of the labs one as well?

Thanks for the offer @Gehel. This cluster in Tool Labs is pretty small and Stashbot and the SAL and bash tools I maintain are the main users of it, so I don't mind being the one to do the upgrade. I'll be in a better position to test things than most people would. I may ask for your help if I run into difficulty in the updates though.

Mentioned in SAL (#wikimedia-cloud) [2017-06-22T16:19:56Z] <bd808> Backed up elasticsearch indexes to personal laptop using elasticdump incase T164842 goes horribly wrong

Mentioned in SAL (#wikimedia-cloud) [2017-06-22T17:03:07Z] <bd808> Rolled back attempt at Elasticsearch upgrade. Indices need to be rebuilt with 2.x before 5.x can be installed. T164842

bd808 added a comment.Jun 22 2017, 5:13 PM

Aborted upgrade because of this error:

Caused by: java.lang.IllegalStateException: The index [[bash/8j-7PJe8TFWSfT5pbViVWg]] was created with version [1.7.1] but the minimum compatible version is [2.0.0-beta1]. It should be re-indexed in Elasticsearch 2.x before upgrading to 5.3.2.

I need to install the migration plugin and find out all the things that need to be fixed before trying again. Once things are ready, the rough steps needed are:

  • sudo -i puppet agent --disable 'upgrading elasticsearch'
  • sudo service elasticsearch stop
  • Set elasticsearch::version: 5 in hiera
  • sudo apt-get install elasticsearch openjdk-8-jdk
  • sudo apt-get purge openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless
  • sudo -i puppet agent --enable
  • sudo -i puppet agent --test --verbose
  • sudo service elasticsearch status and/or sudo journalctl -l --no-pager -u elasticsearch.service
bd808 added a comment.Jun 26 2017, 4:42 PM

Reindexing has been done for the indexes that were created with ElasticSearch <2.x. Migration helper is showing a few issues, but I think they are things that will auto-update or be handled by the Puppet changes that setting elasticsearch::version: 5 does.

Tentatively planning to upgrade the cluster on 2017-06-29 around 21:00Z following the MediaWiki train deploy.

Mentioned in SAL (#wikimedia-cloud) [2017-07-13T21:25:29Z] <bd808> Upgrading ElasticSearch cluster for T164842. There will be service interruptions

Change 365178 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/tools/stashbot@master] Update for ElasticSearch 5.x

https://gerrit.wikimedia.org/r/365178

Change 365178 merged by jenkins-bot:
[labs/tools/stashbot@master] Update for ElasticSearch 5.x

https://gerrit.wikimedia.org/r/365178

Mentioned in SAL (#wikimedia-cloud) [2017-07-13T22:02:33Z] <bd808> Updated to 3c3810c for T164842

bd808 closed this task as Resolved.Jul 14 2017, 12:39 AM