Perform both upgrades on a monday, as both updates will have minor downtime of the logging service and we don't want to lose logs related to the train rolling forward.
Deploy logstash 5.x, around april 10th.
- Take logstash out of the experimental apt repo and upload to main
- merge and git-deploy new logstash plugins
- Merge puppet patch for logstash 5.x configuration
- force a puppet run on logstash100[123]
- apt-get install logstash on logstash100[123]. Unfortunately because there is no LVS balancing writes we may lose a couple things sent to hosts that are being upgraded.
- Check logs are still coming in. Verify no problems recorded in /var/lib/logstash. Note there are some existing errors about Gelfd that are expected (T161563)
Assuming no hiccups, deploy elasticsearch 5.x and kibana 5.x april 24th
- Check that all indices on logstash have been created with elasticsearch >= 2.x. Delete anything old.
- Record the output of curl logstash1001.eqiad.wmnet/_cluster/settings to be re-applied. Evaluate if any of these settings should instead be moved into puppet as permanent settings.
- Merge elasticsearch plugins to operations/software/elasticsearch/plugins
- Pull new elasticsearch plugins to beta cluster
- Install elasticsearch .deb's across cluster
- Shut down elasticsearch on all nodes
- Remount /var/lib/elasticsearch/production-logstash-eqiad to /srv/elasticsearch/production-logstash-eqiad on logstash100[456]
- Can this be done prior to everything else, one node at a time?
- Bring up one logstash data node, make sure everything is happy
- Bring up the rest of the logstash cluster
- Install new kibana deb on logstash100[123]
- Double check logstash installed the newly deployed 5.x versions of index templates. If not manually install logstash and apifeatureusage templates via REST api.