Perform both upgrades on a monday, as both updates will have minor downtime of the logging service and we don't want to lose logs related to the train rolling forward.
Deploy logstash 5.x, around april 10th.
# apt-get install logstash to pull in 5.x. Double check that this wont restart the daemon
# merge and git-deploy new logstash plugins
# Merge puppet patch for logstash 5.x configuration
# force a puppet run on logstash100[123]
# apt-get install logstash on logstash100[123]. Unfortunately because there is no LVS balancing writes we may lose a couple things sent to hosts that are being upgraded.
# Check logs are still coming in. Verify no problems recorded in /var/lib/logstash. Note there are some existing errors about Gelfd that are expected (T161563)
Assuming no hiccups, deploy elasticsearch 5.x and kibana 5.x april 24th
# Check that all indices on logstash have been created with elasticsearch >= 2.x. Delete anything old.
# Record the output of curl search.svc.codfw.wmnet/_cluster/settings to be re-applied
# Merge elasticsearch plugins to operations/software/elasticsearch/plugins
# Pull new elasticsearch plugins to beta cluster
# Install elasticsearch .deb's across cluster
# Shut down elasticsearch on all nodes
# Remount /var/lib/elasticsearch/production-logstash-eqiad to /srv/elasticsearch/production-logstash-eqiad on logstash100[456]
#* Can this be done prior to everything else, one node at a time?
# Bring up one logstash data node, make sure everything is happy
# Bring up the rest of the logstash cluster
# Install new kibana deb on logstash100[123]
# Double check logstash installed the newly deployed 5.x versions of index templates. If not manually install logstash and apifeatureusage templates via REST api.