@bd808 might be interested in this one as well. Note that our logstash cluster is already upgraded to elasticsearch 2.3.5. We are planning to upgrade the cirrus clusters to elasticsearch 5.x as well. It might make sense to coordinate all those upgrade and move to 5.x everywhere for logstash and elasticsearch.

The Logstash Forwarder and Lumberjack inputs are not used in WMF production and thus not a concern for upgrades. We are already running Elasticsearch 2.3.5 as the backend as @Gehel has pointed out. We did this without a Logstash upgrade with no major issues. I would rather not mess with any of this until the Discovery-ARCHIVED team moves us to Elasticsearch 5.x.

For completeness, the upgrade to elasticsearch 5.x is tracked on T154501

Does this task supersede T127676?

Yes it does, i've declined the 2.x task

Some initial thoughts:

• It would be nice to not upgrade logstash, elasticsearch and kibana all in lockstep. I worry that upgrading all three components will make it harder to figure out where new problems come from. Although perhaps I'm worrying too much
• Could test the latest logstash against 2.3.5, it might generally be ok. Not sure what features are only available in 2.4 vs 2.3.
• The 5.x series of logstash is compatible with elasticsearch 2.4.x and 5.x . Unfortunately we are running 2.3.5 in production. The upgrade from 2.3.5 to 2.4.4 is likely pretty painless though, after the prod search clusters are moved to 5.x could perhaps upgrade these to 2.4.4?
• kibana versioning is pretty much in lockstep with elasticsearch. kibana 5.2 is only supported against elasticsearch 5.2. Same story for 5.1 and 5.0.

Actually i was mis-reading the compatability matrix. For elasticsearch 2.3.x logstash is reported as being compatible with 2.0.x - 5.2.x. Which is a bit funny because we are still using logstash 1.5.3. in prod. With elasticsearch upgraded to 5.2.x logstash support will be 2.4.x to 5.2.x

@EBernhardson thinks this may need splitting up into several tasks. There's a lot of major version upgrades for the entire ELK stack, and some of them are cross-dependent. First he will get this running in vagrant, then beta cluster, then putting in production and seeing if it breaks things.

Change 342772 had a related patch set uploaded (by EBernhardson):
[mediawiki/vagrant] [WIP] Update logstash to 5.2.2

https://gerrit.wikimedia.org/r/342772

Pulled the production .kibana index into my local vagrant and tested, looks like all the visualizations "just work". Going to wrap up the vagrant patch, and get the patch for labs ready to upgrade that cluster.

Change 344964 had a related patch set uploaded (by EBernhardson):
[operations/puppet@production] [WIP] Upgrade logstash to 5.x

https://gerrit.wikimedia.org/r/344964

Change 344965 had a related patch set uploaded (by EBernhardson):
[operations/puppet@production] [WIP] Update elk stack to 5.x

https://gerrit.wikimedia.org/r/344965

Logstash has been upgraded to 5.x on the beta cluster. Everything is prepped for elasticsearch and kibana to upgrade as well, but will leave the logstash upgrade running on its own for a few days or a week to isolate any problems related explicitly to this upgrade and not a confluence of changes due to three upgrades at once.

Change 342772 merged by jenkins-bot:
[mediawiki/vagrant@master] Update logstash to 5.2.2, kibana to 5.1.2

https://gerrit.wikimedia.org/r/342772

Change 344964 merged by Gehel:
[operations/puppet@production] Upgrade logstash to 5.x

https://gerrit.wikimedia.org/r/344964

this has been deployed and will need to wait until after the next deployment freeze to flip the switch.

Change 344965 merged by Gehel:
[operations/puppet@production] Update elk stack to 5.x

https://gerrit.wikimedia.org/r/344965