Page MenuHomePhabricator

Update logstash on wikimedia to 5.x
Closed, ResolvedPublic

Description

This is a feature request but also a bug fix request too.

The current log stash version wikimedia uses is really old + it has a bug in the one in wikimedia uses where sometimes ssl will not work properly.

See https://www.elastic.co/blog/logstash-1-5-4-and-1-4-5-released

"Reverted a change in our harden SSL fix, that prevented Logstash Forwarder and Lumberjack output clients to connect to 1.5.3 instances (#3657)"

+ Logstash 2.x has more default plugins but some of those plugins are removed as default in 5.x.

See https://www.elastic.co/blog/logstash-2-0-0-released and https://www.elastic.co/blog/logstash-5-0-0-released please

This will also add support for elasticsearch 2.x

Event Timeline

Paladox created this task.Jan 3 2017, 11:31 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2017, 11:31 AM
Paladox updated the task description. (Show Details)
Restricted Application added projects: Discovery, Discovery-Search. · View Herald TranscriptJan 3 2017, 11:34 AM
Paladox updated the task description. (Show Details)Jan 3 2017, 11:38 AM
Paladox updated the task description. (Show Details)Jan 3 2017, 11:41 AM

@bd808 might be interested in this one as well. Note that our logstash cluster is already upgraded to elasticsearch 2.3.5. We are planning to upgrade the cirrus clusters to elasticsearch 5.x as well. It might make sense to coordinate all those upgrade and move to 5.x everywhere for logstash and elasticsearch.

demon removed a subscriber: demon.Jan 3 2017, 4:03 PM
Dzahn removed a subscriber: Dzahn.Jan 3 2017, 4:43 PM
bd808 added a comment.Jan 3 2017, 6:17 PM

The Logstash Forwarder and Lumberjack inputs are not used in WMF production and thus not a concern for upgrades. We are already running Elasticsearch 2.3.5 as the backend as @Gehel has pointed out. We did this without a Logstash upgrade with no major issues. I would rather not mess with any of this until the Discovery team moves us to Elasticsearch 5.x.

Gehel added a comment.Jan 4 2017, 10:05 AM

For completeness, the upgrade to elasticsearch 5.x is tracked on T154501

Deskana renamed this task from Update logstash on wikimedia to 2.x or 5.x to Update logstash on wikimedia to 5.x.Jan 4 2017, 5:03 PM
Deskana moved this task from needs triage to Current work on the Discovery-Search board.
Deskana added a subscriber: Deskana.

I will assume we'd be going with 5.x to keep Logstash at the same version as other things in our ELK stack.

Deskana triaged this task as Normal priority.Jan 4 2017, 5:04 PM

Does this task supersede T127676?

Yes it does, i've declined the 2.x task

Some initial thoughts:

  • It would be nice to not upgrade logstash, elasticsearch and kibana all in lockstep. I worry that upgrading all three components will make it harder to figure out where new problems come from. Although perhaps I'm worrying too much
  • Could test the latest logstash against 2.3.5, it might generally be ok. Not sure what features are only available in 2.4 vs 2.3.
  • The 5.x series of logstash is compatible with elasticsearch 2.4.x and 5.x . Unfortunately we are running 2.3.5 in production. The upgrade from 2.3.5 to 2.4.4 is likely pretty painless though, after the prod search clusters are moved to 5.x could perhaps upgrade these to 2.4.4?
  • kibana versioning is pretty much in lockstep with elasticsearch. kibana 5.2 is only supported against elasticsearch 5.2. Same story for 5.1 and 5.0.
EBernhardson added a comment.EditedMar 13 2017, 9:59 PM

Actually i was mis-reading the compatability matrix. For elasticsearch 2.3.x logstash is reported as being compatible with 2.0.x - 5.2.x. Which is a bit funny because we are still using logstash 1.5.3. in prod. With elasticsearch upgraded to 5.2.x logstash support will be 2.4.x to 5.2.x

@EBernhardson thinks this may need splitting up into several tasks. There's a lot of major version upgrades for the entire ELK stack, and some of them are cross-dependent. First he will get this running in vagrant, then beta cluster, then putting in production and seeing if it breaks things.

Change 342772 had a related patch set uploaded (by EBernhardson):
[mediawiki/vagrant] [WIP] Update logstash to 5.2.2

https://gerrit.wikimedia.org/r/342772

Pulled the production .kibana index into my local vagrant and tested, looks like all the visualizations "just work". Going to wrap up the vagrant patch, and get the patch for labs ready to upgrade that cluster.

Change 344964 had a related patch set uploaded (by EBernhardson):
[operations/puppet@production] [WIP] Upgrade logstash to 5.x

https://gerrit.wikimedia.org/r/344964

Change 344965 had a related patch set uploaded (by EBernhardson):
[operations/puppet@production] [WIP] Update elk stack to 5.x

https://gerrit.wikimedia.org/r/344965

Logstash has been upgraded to 5.x on the beta cluster. Everything is prepped for elasticsearch and kibana to upgrade as well, but will leave the logstash upgrade running on its own for a few days or a week to isolate any problems related explicitly to this upgrade and not a confluence of changes due to three upgrades at once.

Change 342772 merged by jenkins-bot:
[mediawiki/vagrant@master] Update logstash to 5.2.2, kibana to 5.1.2

https://gerrit.wikimedia.org/r/342772

Change 344964 merged by Gehel:
[operations/puppet@production] Upgrade logstash to 5.x

https://gerrit.wikimedia.org/r/344964

debt added a subscriber: debt.Apr 11 2017, 5:05 PM

this has been deployed and will need to wait until after the next deployment freeze to flip the switch.

Change 344965 merged by Gehel:
[operations/puppet@production] Update elk stack to 5.x

https://gerrit.wikimedia.org/r/344965

debt closed this task as Resolved.May 30 2017, 5:18 PM