Page MenuHomePhabricator

Upgrade Tool Labs elasticsearch to 5.x
Closed, ResolvedPublic


Follow upgrade path of production Elasticsearch clusters by upgrading to the 5.x version in Wikimedia's apt repo.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@bd808 since I'm upgrading the cirrus clusters, do you want me to take care of the labs one as well?

@bd808 since I'm upgrading the cirrus clusters, do you want me to take care of the labs one as well?

Thanks for the offer @Gehel. This cluster in Tool Labs is pretty small and Stashbot and the SAL and bash tools I maintain are the main users of it, so I don't mind being the one to do the upgrade. I'll be in a better position to test things than most people would. I may ask for your help if I run into difficulty in the updates though.

Mentioned in SAL (#wikimedia-cloud) [2017-06-22T16:19:56Z] <bd808> Backed up elasticsearch indexes to personal laptop using elasticdump incase T164842 goes horribly wrong

Mentioned in SAL (#wikimedia-cloud) [2017-06-22T17:03:07Z] <bd808> Rolled back attempt at Elasticsearch upgrade. Indices need to be rebuilt with 2.x before 5.x can be installed. T164842

Aborted upgrade because of this error:

Caused by: java.lang.IllegalStateException: The index [[bash/8j-7PJe8TFWSfT5pbViVWg]] was created with version [1.7.1] but the minimum compatible version is [2.0.0-beta1]. It should be re-indexed in Elasticsearch 2.x before upgrading to 5.3.2.

I need to install the migration plugin and find out all the things that need to be fixed before trying again. Once things are ready, the rough steps needed are:

  • sudo -i puppet agent --disable 'upgrading elasticsearch'
  • sudo service elasticsearch stop
  • Set elasticsearch::version: 5 in hiera
  • sudo apt-get install elasticsearch openjdk-8-jdk
  • sudo apt-get purge openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless
  • sudo -i puppet agent --enable
  • sudo -i puppet agent --test --verbose
  • sudo service elasticsearch status and/or sudo journalctl -l --no-pager -u elasticsearch.service

Reindexing has been done for the indexes that were created with ElasticSearch <2.x. Migration helper is showing a few issues, but I think they are things that will auto-update or be handled by the Puppet changes that setting elasticsearch::version: 5 does.

Tentatively planning to upgrade the cluster on 2017-06-29 around 21:00Z following the MediaWiki train deploy.

Mentioned in SAL (#wikimedia-cloud) [2017-07-13T21:25:29Z] <bd808> Upgrading ElasticSearch cluster for T164842. There will be service interruptions

Change 365178 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/tools/stashbot@master] Update for ElasticSearch 5.x

Change 365178 merged by jenkins-bot:
[labs/tools/stashbot@master] Update for ElasticSearch 5.x