Page MenuHomePhabricator

Ship MX logs to ELK
Open, MediumPublic

Description

Let's ship mail logs from the MX servers to ELK.

There we can build visualizations, monitor trends, report on envelope details, etc.

Event Timeline

herron created this task.Jun 13 2018, 6:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 13 2018, 6:53 PM
faidon added a subscriber: faidon.Jun 14 2018, 3:52 PM

Our email logs can be pretty sensitive, especially since they include our corporate emails passing through (senders, recipients, timestamps etc.).

So, I'm a bit concerned about both whether our access controls are sufficient, and also whether the security of this platform is good enough to protect us against privilege escalation attacks.

Is this something that has been considered already? I'm not objecting this per se -- I'd just like for this to be carefully planned and reviewed because of the sensitivity of this particular log stream :)

Joe triaged this task as Medium priority.Jun 18 2018, 2:49 PM

That makes sense, and likely applies to other types of logs as well. It seems to me like something worth solving in the ELK stack.

Do we have an X-Pack subscription from elastic? If so we could use that to configure role based acls and segment access to various types of logs based on ldap group membership natively in Elasticsearch and Kibana.

If not, another potential approach is to...

  • Ship this type of sensitive log to an index prefix that is outside the scope of the currently configured Kibana search prefix.
  • Restrict the ability of users to modify the Kibana index prefix setting via the web interface.
  • Stand up a "root only" Kibana instance with strict authorization (require cn=ops,ou=groups,dc=wikimedia,dc=org or such) that is configured to search across both root and non-root indices.

Today we have Logstash outputting to Elasticsearch indices prefixed with logstash- and Kibana is configured to query logstash-*. We could change the prefix something like logstash-all- for unprivileged logs, and logstash-root- for "root only" logs.

IIRC each Kibana instance has it's own ES index for config storage, meaning visualizations, dashboards, etc would not kept in sync between two instances with differing search prefix configs. So that's a compromise to think about when separating log access this way. Maybe there's a way around it. Worst case these could be exported/imported between instances.

Vvjjkkii renamed this task from Ship MX logs to ELK to a2aaaaaaaa.Jul 1 2018, 1:04 AM
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from a2aaaaaaaa to Ship MX logs to ELK.Jul 2 2018, 12:21 PM
CommunityTechBot lowered the priority of this task from High to Medium.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.
herron removed a project: Mail.Nov 9 2018, 9:50 PM