Mentioned in SAL [2016-02-25T00:57:55Z] <bd808> Started crashed Logstash process on logstash1002 (systemd doesn't restart authomatically due to T127677)

Maybe some nice soul from SRE can give me tips on how to make a custom unit file to put in our Puppet config?

I wrote an systemd unit based on the current init script. It's totally untested, though!
https://phabricator.wikimedia.org/P2671

Some notes:

• In the sysvinit script stdout and stderr were redirected to logfiles, that's intentionally not supported with the systemd directives: https://lists.freedesktop.org/archives/systemd-devel/2012-March/004703.html Instead it's here sent to journalctl here, which is quite convenient. But we can also retain the old behaviour with amending the ExecStart line
• The sysv init script nices the logstash daemon to 19, we can do the same with "Nice=19", but I left it out since it seemed somehow cargo cult. Is that really needed?
• systemd has some nice features to restrict the logstash daemon (in case the process gets compromised e.g.). We can add these once the basic service is running fine.

In T127677#2062644, @MoritzMuehlenhoff wrote:

I wrote an systemd unit based on the current init script. It's totally untested, though!
https://phabricator.wikimedia.org/P2671

Cool! Let's test it out in the beta cluster install. We can do that by turning your paste into a proper Puppet patch and cherry-picking that on deployment-puppetmaster.

Some notes:

• In the sysvinit script stdout and stderr were redirected to logfiles, that's intentionally not supported with the systemd directives: https://lists.freedesktop.org/archives/systemd-devel/2012-March/004703.html Instead it's here sent to journalctl here, which is quite convenient. But we can also retain the old behaviour with amending the ExecStart line

As long as we can review the log output I can live without /var/log/logstash files. I think something like journalctl -f -u logstash would cover most of what I've done with the on disk logs from Logstash.

• The sysv init script nices the logstash daemon to 19, we can do the same with "Nice=19", but I left it out since it seemed somehow cargo cult. Is that really needed?

I bet that is in the default scripts for installs where Logstash is being used as a log shipper service on a host that is primarily doing some other work. For our use cases I don't see the point in renicing.

• systemd has some nice features to restrict the logstash daemon (in case the process gets compromised e.g.). We can add these once the basic service is running fine.

Sounds like a good plan. Implement, inspect and adapt. :)

2x logstashes (1001 + 1002) pretty much simultaneously crashed today with errors like:

Error: Your application used more memory than the safety cap of 500M.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

The limit is configurable via LS_HEAP_SIZE:
https://discuss.elastic.co/t/is-it-possible-to-give-ls-heap-size-xms-and-xmx-while-starting-logstash-1-4-2-or-15-2-like-elasticsearch/39493

https://gerrit.wikimedia.org/r/#/c/274696/ is running on deployment-logstash2 for a while now