When provisioning the new jessie based Elasticsearch backends for the production Logstash cluster I had to manually curl http://apt.wikimedia.org/wikimedia/pool/thirdparty/e/elasticsearch/elasticsearch_1.3.6_all.deb and install the deb with dpkg (T97645). We should get our local apt repo updated to include the more modern Elasticsearch debs in the jessie index.
Needed debs:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | bd808 | T69817 Monitor for anomalies/spikes in read failures of memcached | |||
| Resolved | bd808 | T100735 Have Logstash report per-channel log message rate to Graphite | |||
| Resolved | bd808 | T99735 Upgrade Logstash to 1.5.3 | |||
| Resolved | bd808 | T97545 reinstall logstash1001-1003 | |||
| Resolved | MoritzMuehlenhoff | T98042 Update Wikimedia apt repo to include debs for Elasticsearch & Logstash on jessie |
Having read the comments on T97645, do you want us to just reuse the elasticsearch_1.3.6_all.deb that you manually installed, or are there better packages elsewhere? I had little luck trying to hunt around on packages.elasticsearch.org.
@ArielGlenn, If we locked the version of Elasticsearch that cirrus used then it'd be fine to just include them all. Right now its just ensure => present.
Pinning the version in Puppet is problematic for graceful cluster upgrades isn't it? That would require you to disable Puppet while upgrading the cluster, upgrade, adjust Puppet and then re-enable.
We would probably want to at least move the version to a hiera variable so that that we weren't forced to upgrade all of the various Elasticsearch clusters controlled with Puppet in lock step.
Today, having the current 1.3.6 that is in out thirdparty pool available on jessie would make me happy. We probably need to think about our future upgrade process though. For the Logstash related clusters in prod and beta I'd love to keep lagging a bit behind Nik's work on the Cirrus clusters to minimize the chance of surprises when bumping Elasticsearch versions. Unstable log aggregation can be worse than no log aggregation.
The version in standard jessie in 1.0.2, but there's 1.4.5 in experimental which could easily be backported, the current debs ship binary jars provided by upstream.
1.3.6 was the version I installed on the Jessie boxes for Logstash because that matches the version that is installed on the Precise Logstash hosts. Putting 1.3.9 in apt instead of 1.3.6 would be fine. I can do a cluster update for all of them. We have been holding off on newer versions waiting for 1.6.0 to ship upstream.
The current cluster has been updated to elasticsearch_1.3.9_all.deb now so this is the version we should make the default for jessie. I just downloaded the deb from http://apt.wikimedia.org/wikimedia/pool/thirdparty/e/elasticsearch/elasticsearch_1.3.9_all.deb and installed it with dpkg to do the upgrade.
Elasticsearch 1.6.0 should be the default jessie deb now that it the default for precise/trusty.
With T106126, we should be importing https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.0.deb for jessie
1.7.1 has been released, which fixes a data-loss bug, I'll import that version as well:
"The bug in question (#12487) can, in very rare circumstances involving multiple simultaneous node failures or restarts, cause all copies of a shard to be deleted from the cluster. This bug was introduced in 1.5.0."
https://www.elastic.co/blog/elasticsearch-1-7-1-and-1-6-2-released