Planning to deploy on 2015-08-11 immediately after T99735: Upgrade Logstash to 1.5.3 is complete.

Schedule icinga 3 hour maintenance window before starting (needs SRE help)

Deploy plan for each of logstash100[1-6]:

#!/usr/bin/env bash

# Stop cluster from reallocating shards
curl -s -XPUT localhost:9200/_cluster/settings?pretty -d '{
  "transient" : {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

# Upgrade to latest openjdk packages
sudo apt-get install openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless

# Install new ES package
sudo service elasticsearch stop
sudo apt-get --assume-yes install elasticsearch
sudo service elasticsearch start

# Wait for service to restart
until curl -s localhost:9200/_cluster/health?pretty; do
    sleep 1
done

# Re-enable shard allocation
curl -s -XPUT localhost:9200/_cluster/settings?pretty -d '{
  "transient" : {
    "cluster.routing.allocation.enable": "all"
  }
}'

# Wait for the cluster to recover
until curl -s localhost:9200/_cluster/health?pretty |
    tee /tmp/health |
    grep green; do
    cat /tmp/health
    sleep 1
done

(edited to add openjdk update)

In T108040#1525078, @bd808 wrote:

Planning to deploy on 2015-08-11 immediately after T99735: Upgrade Logstash to 1.5.3 is complete.

Schedule icinga 3 hour maintenance window before starting (needs SRE help)

Deploy plan for each of logstash100[1-6]:

#!/usr/bin/env bash

# Stop cluster from reallocating shards
curl -s -XPUT localhost:9200/_cluster/settings?pretty -d '{
  "transient" : {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

# Upgrade to latest openjdk packages
sudo apt-get install openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless

# Install new ES package
sudo service elasticsearch stop
sudo apt-get --assume-yes install elasticsearch
sudo service elasticsearch start

# Wait for service to restart
until curl -s localhost:9200/_cluster/health?pretty; do
    sleep 1
done

# Re-enable shard allocation
curl -s -XPUT localhost:9200/_cluster/settings?pretty -d '{
  "transient" : {
    "cluster.routing.allocation.enable": "all"
  }
}'

# Wait for the cluster to recover
until curl -s localhost:9200/_cluster/health?pretty |
    tee /tmp/health |
    grep green; do
    cat /tmp/health
    sleep 1
done

(edited to add openjdk update)

LGTM, did you look at es-tool by any chance? it should be able to provide some of the above

In T108040#1526786, @fgiunchedi wrote:

LGTM, did you look at es-tool by any chance? it should be able to provide some of the above

es-tool is only provisioned on trusty hosts; the logstash cluster is all jessie. :(