There seem to be some correlation with a high number of failed relocations that happened just before mjolnir failing (see [[ URL | logstash ]]). No idea if there is a causality here.

Mentioned in SAL (#wikimedia-operations) [2018-08-17T16:24:48Z] <gehel> disabling systemd state check for elastic eqiad until T202120 is fixed

Looking at elastic1020 we have in journalctl -u mjolnir-kafka-bulk-daemon

Aug 17 16:14:14 elastic1020 systemd[1]: mjolnir-kafka-bulk-daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 16:14:14 elastic1020 systemd[1]: mjolnir-kafka-bulk-daemon.service: Unit entered failed state.
Aug 17 16:14:14 elastic1020 systemd[1]: mjolnir-kafka-bulk-daemon.service: Failed with result 'exit-code'.
Aug 17 16:33:53 elastic1020 systemd[1]: Started MjoLniR kafka bulk daemon.

There is a matching puppet log in journalctl for puppet:

Aug 17 16:33:53 elastic1020 puppet-agent[9782]: (/Stage[main]/Profile::Mjolnir::Kafka_bulk_daemon/Systemd::Service[mjolnir-kafka-bulk-daemon]/Service[mjolnir-kafka-bulk-daemon]/ensure

This suggests systemd isn't restarting the service, and we should set Restart=always in the systemd config to match how the service expects to run.

Change 453450 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Mjolnir daemons should run with Restart=always

https://gerrit.wikimedia.org/r/453450

Change 453450 merged by Gehel:
[operations/puppet@production] Mjolnir daemons should run with Restart=always

https://gerrit.wikimedia.org/r/453450

Restart=always on the systemd unit should fix the immediate issue. This has been deployed. I'm keeping this task open for a few more days, until we can validate that the issue is not reproduced.

A full export ran over the daemon from 8/20 13:24 to 8/22 02:00 without triggering this issue again. I think it can be closed.