Page MenuHomePhabricator

Investigate approaches to ingest sensitive log producers
Closed, ResolvedPublic

Description

This task tracks the potential solutions to handle sensitive log producers. This requirement stems from the fact that certain categories of logs can be considered sensitive and their access should be restricted to root-level users only.

Out of the box Kibana/Elasticsearch do not support properly isolated multi tenancy, thus the need to investigate possible solutions.


Possible Approaches:

  1. "Stack" formerly "X-Pack" from elastic. (https://www.elastic.co/products/stack)
    • Pros
      • "Official" solution provided by Elastic which provides rbac by way of Elasticsearch and Kibana plugins (among other non-security features).
    • Cons
  2. Open Distro for Elasticsearch https://opendistro.github.io/for-elasticsearch/
    • Pros
      • Provides RBAC security features comparable to X-pack security without elastic licensing fees
      • Apache 2.0 Licensed
    • Cons
      • Adds some complexity to the Kibana and Elasticsearch deployment in that plugins and dependencies must to be installed/configured.
  3. Search Guard community edition (https://search-guard.com/product/)
    • Pros
      • Adds rbac to Elasticsearch and Kibana by way of apache licensed Elasticsearch and Kibana plugins (https://github.com/floragunncom) (with enterprise upgrade offering more advanced features)
    • Cons
      • Does not natively support LDAP at free "community" tier, would need to work around this.
      • Adds some complexity to the Kibana and Elasticsearch deployment in that plugins and dependencies must to be installed/configured.
  4. Implement our own basic access controls with an authenticating proxy filtering requests to/from Kibana and Elasticsearch.
    • Pros
      • "Drop-in" approach that does not require substantial reconfiguration of Elasticsearch or Kibana.
    • Cons
      • Simple URI filtering on the frontend reverse proxy could restrict results displayed by Kibana UI, but clever users could craft requests to retrieve sensitive log data without using Kibana.
      • An Elasticsearch filtering proxy would need to inspect/modify request bodies of many API calls, which would be fragile and difficult to maintain/troubleshoot.
      • Separate Kibana config indices are needed (one per role), requiring a mechanism to sync visualizations/dashboards/etc. between instances.
  5. Deploy additional Elasticsearch cluster (either on shared, or separate hardware) and additional Kibana instance for sensitive log data
    • Pros
      • Simple approach from a tooling perspective, effectively duplicating existing architecture (without requirement of additional hosts) and adding output rules to route logs to the appropriate cluster.
    • Cons
      • Duplicates infrastructure maintenance overhead
      • Separate Kibana config indices are needed (one per role), requiring a mechanism to sync visualizations/dashboards/etc. between instances.

Event Timeline

herron triaged this task as Medium priority.Oct 2 2018, 5:26 PM
herron updated the task description. (Show Details)Dec 4 2018, 7:05 PM
herron updated the task description. (Show Details)Dec 4 2018, 7:08 PM
herron updated the task description. (Show Details)Dec 4 2018, 7:16 PM
herron added a subscriber: herron.Dec 4 2018, 7:19 PM

Updated the description to outline several possible approaches. To me option #2 stands out as worthy of a POC.

herron updated the task description. (Show Details)Dec 5 2018, 4:13 PM
herron updated the task description. (Show Details)Dec 10 2018, 4:43 PM
fgiunchedi closed this task as Resolved.Jan 16 2019, 11:17 AM
fgiunchedi claimed this task.

One of these approaches will be implemented as part of stretch goals of T213157: Increase utilization of application logging pipeline (FY2018-2019 Q3 TEC6)

https://opendistro.github.io/for-elasticsearch/ appears to be a valid option, although this was resolved I'll update the description to include it

herron updated the task description. (Show Details)Sep 13 2019, 6:11 PM