Page MenuHomePhabricator

Upgrade Logstash Elasticsearch cluster to 1.7.1
Closed, ResolvedPublic

Description

Update to the latest Elasticsearch bug fix release.

Related Objects

Event Timeline

bd808 claimed this task.
bd808 raised the priority of this task from to Medium.
bd808 updated the task description. (Show Details)
bd808 added a subscriber: bd808.

Planning to deploy on 2015-08-11 immediately after T99735: Upgrade Logstash to 1.5.3 is complete.

Schedule icinga 3 hour maintenance window before starting (needs SRE help)

Deploy plan for each of logstash100[1-6]:

#!/usr/bin/env bash

# Stop cluster from reallocating shards
curl -s -XPUT localhost:9200/_cluster/settings?pretty -d '{
  "transient" : {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

# Upgrade to latest openjdk packages
sudo apt-get install openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless

# Install new ES package
sudo service elasticsearch stop
sudo apt-get --assume-yes install elasticsearch
sudo service elasticsearch start

# Wait for service to restart
until curl -s localhost:9200/_cluster/health?pretty; do
    sleep 1
done

# Re-enable shard allocation
curl -s -XPUT localhost:9200/_cluster/settings?pretty -d '{
  "transient" : {
    "cluster.routing.allocation.enable": "all"
  }
}'

# Wait for the cluster to recover
until curl -s localhost:9200/_cluster/health?pretty |
    tee /tmp/health |
    grep green; do
    cat /tmp/health
    sleep 1
done

(edited to add openjdk update)

Planning to deploy on 2015-08-11 immediately after T99735: Upgrade Logstash to 1.5.3 is complete.

Schedule icinga 3 hour maintenance window before starting (needs SRE help)

Deploy plan for each of logstash100[1-6]:

#!/usr/bin/env bash

# Stop cluster from reallocating shards
curl -s -XPUT localhost:9200/_cluster/settings?pretty -d '{
  "transient" : {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

# Upgrade to latest openjdk packages
sudo apt-get install openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless

# Install new ES package
sudo service elasticsearch stop
sudo apt-get --assume-yes install elasticsearch
sudo service elasticsearch start

# Wait for service to restart
until curl -s localhost:9200/_cluster/health?pretty; do
    sleep 1
done

# Re-enable shard allocation
curl -s -XPUT localhost:9200/_cluster/settings?pretty -d '{
  "transient" : {
    "cluster.routing.allocation.enable": "all"
  }
}'

# Wait for the cluster to recover
until curl -s localhost:9200/_cluster/health?pretty |
    tee /tmp/health |
    grep green; do
    cat /tmp/health
    sleep 1
done

(edited to add openjdk update)

LGTM, did you look at es-tool by any chance? it should be able to provide some of the above

LGTM, did you look at es-tool by any chance? it should be able to provide some of the above

es-tool is only provisioned on trusty hosts; the logstash cluster is all jessie. :(

bd808 moved this task from In Dev/Progress to Done on the User-bd808 board.