• several system log candidates are listed on T63779
  • T62689: Add salt logs to logstash
  • T64177: Add localisation update logs to logstash
  • T63782: Add varnish logs to logstash
  • T63780: Add swift logs to logstash
• T6: Get !log entries from #-operations into logstash would be low volume but useful.
• T64505: Add gerrit event stream events to logstash

From past experience, once you start aggregating logs in a useful way you will continually find more event sources to add. I think we should plan on a system that can be grown horizontally as we find more things we want to do with it. Luckily, this should be pretty easy. Any number of Logstash agents can be used to feed data to the backing Elasticsearch cluster and Elasticsearch itself is designed to scale by adding nodes and increasing shard count on indices.

Separating Elasticsearch from Logstash + redis + apache would be a good start. The existing misc boxes should be able to handle ingesting a lot of log traffic if relieved from their duty of indexing the resulting documents and delivering search results.

Here's my proposal:

• Add 3 hosts matching the machine spec used for the MediaWiki Elasticsearch cluster
• Move the Logstash Elasticsearch backend to those hosts
• Keep the current logstash100[123] boxes to run the rest of the stack (Apache + Redis + Logstash)

Scale the system by:

• Adding Elasticsearch nodes if and when Elasticsearch becoes a bottleneck on the 3 node cluster (shard daily indices across more than 1 shard, distribute across more hosts)
• Separating Redis and Logstash to separate machines if there is cpu or ram contention on those host between the two processes
• Add more hosts running logstash if and when needed due to cpu consumed by logstash processing

Unanswered questions:

• Do we need Logstash processing cluster in codfw or can we ship logs direct to eqiad?
• Do we need Logstash Elasticsearch cluster in codfw as either replica of combined data from both DCs or as a codfw only log storage system?
• Do we need a separate "ops only" Elasticsearch cluster for log storage or can we redact or partition the data in the same cluster.

In T84958#988453, @bd808 wrote:

Here's my proposal:

• Add 3 hosts matching the machine spec used for the MediaWiki Elasticsearch cluster

Does it make sense to use spinning disks rather than SSDs? High indexing rate loves SSDs but keeping lots of data around likes the bigger rotating disks. What about RAM?

In T84958#988487, @Manybubbles wrote:

In T84958#988453, @bd808 wrote:

Here's my proposal:

• Add 3 hosts matching the machine spec used for the MediaWiki Elasticsearch cluster

Does it make sense to use spinning disks rather than SSDs? High indexing rate loves SSDs but keeping lots of data around likes the bigger rotating disks. What about RAM?

At this moment, the base size of the data set is 990G for the last 30 days. With the growth that has been seen as a result of the MediaWiki conversion to Monolog+Redis by mid-month in February that base storage may grow to something like 1.5T. I would like to see each index have 2 replicas (3 total copies) available across the backing Elasticsearch cluster so with 3 nodes we would want 1.5T plus a healthy growth factor available for index storage.

I personally think we will be fine with spinning disks for IOPS. If at some point in the growth of the logstash cluster we really need to be able to ingest messages at a rate where disk IOPS become a problem we can start sharding the daily index and if we get to a point where that doesn't help we can add a set of nodes with SSD to hold the current day's index and then use index routing in Elastcisearch itself to move the index from nodes using SSD to nodes using spinning disks each day.

Ram I think is a more the merrier thing. If we are using a 30G fixed heap then we should have 45-60G available for the JVM plus file system cache for the index files plus more for OS overhead.

There is currently ongoing discussion on https://rt.wikimedia.org/Ticket/Display.html?id=9199 in regards to the disks being SATA or SAS.

The order for these systems has been placed on https://rt.wikimedia.org/Ticket/Display.html?id=9199. The ETA as of now is 2015-04-24 (Friday) for delivery. We'll need a day or two to receive them in and get them racked and ready for installation & then installed. I wouldn't expect to see these ready for service implementation sooner than the 30th. (It could be faster of course, but I don't like to promise too short a timeline.)