Follow upgrade path of production Elasticsearch clusters by upgrading to the 5.x version in Wikimedia's apt repo.
Thanks for the offer @Gehel. This cluster in Tool Labs is pretty small and Stashbot and the SAL and bash tools I maintain are the main users of it, so I don't mind being the one to do the upgrade. I'll be in a better position to test things than most people would. I may ask for your help if I run into difficulty in the updates though.
Aborted upgrade because of this error:
Caused by: java.lang.IllegalStateException: The index [[bash/8j-7PJe8TFWSfT5pbViVWg]] was created with version [1.7.1] but the minimum compatible version is [2.0.0-beta1]. It should be re-indexed in Elasticsearch 2.x before upgrading to 5.3.2.
I need to install the migration plugin and find out all the things that need to be fixed before trying again. Once things are ready, the rough steps needed are:
- sudo -i puppet agent --disable 'upgrading elasticsearch'
- sudo service elasticsearch stop
- Set elasticsearch::version: 5 in hiera
- sudo apt-get install elasticsearch openjdk-8-jdk
- sudo apt-get purge openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless
- sudo -i puppet agent --enable
- sudo -i puppet agent --test --verbose
- sudo service elasticsearch status and/or sudo journalctl -l --no-pager -u elasticsearch.service
Reindexing has been done for the indexes that were created with ElasticSearch <2.x. Migration helper is showing a few issues, but I think they are things that will auto-update or be handled by the Puppet changes that setting elasticsearch::version: 5 does.
Tentatively planning to upgrade the cluster on 2017-06-29 around 21:00Z following the MediaWiki train deploy.