Page MenuHomePhabricator

Secure inter-datacenter web request log (Kafka) traffic
Closed, ResolvedPublic

Description

The Kafka development team seems to have only begun discussing this recently, in 3Q 2014:
https://cwiki.apache.org/confluence/display/KAFKA/Security

To secure this traffic using IPsec would mean defining SAs on Varnish nodes (Kafka producers) in ESAMS and ULSFO, as well as on the Kafka brokers. IPsec would not be required between Kafka brokers and consumers (Hadoop workers) because at this time that traffic flow is strictly intra-datacenter at EQIAD.

Implementation would involve modifying the ipsec role to reference Hiera keys which enumerate Varnish nodes (including misc and parsoid caches), and extending enumeration of remote IPsec hosts on Varnish nodes to include Kafka brokers.

Event Timeline

Gage raised the priority of this task from to Needs Triage.
Gage updated the task description. (Show Details)
Gage added subscribers: Gage, Ottomata.
Gage triaged this task as Medium priority.Mar 30 2015, 4:05 PM
Gage set Security to None.

FYI, our Kafka upgrade and expansion is complete. All Kafka brokers are now Jessie, so I think this can proceed.

Kafka upstream is getting TLS/SSL support pretty soon. Last rumour I heard is 0.8.3 (due in October) for Kafka proper. librdkafka already has support for it in master which will be released with the next release I think. It should also require no modifications to varnishkafka. I'm not sure what's the status with other libraries we use or may want to use in the future, especially if they're not layered on top of librdkafka.

I don't think it makes much sense to go for IPsec at this point — let's just go for TLS directly.

Due to scheduling constraints and the new-ness of the TLS-capable release of kafka, we're going to rely on IPSec for this for now and revisit TLS for it later. Patches will be incoming this week.

Change 259240 had a related patch set uploaded (by BBlack):
apply ipsec associations to kafka brokers

https://gerrit.wikimedia.org/r/259240

Change 259240 merged by BBlack:
apply ipsec associations to kafka brokers

https://gerrit.wikimedia.org/r/259240

Change 259263 had a related patch set uploaded (by BBlack):
ipsec-for-kafka: limit to kafka1012 cp4011 for testing

https://gerrit.wikimedia.org/r/259263

Change 259263 merged by BBlack:
ipsec-for-kafka: limit to kafka1012 cp4011 for testing

https://gerrit.wikimedia.org/r/259263

Change 259280 had a related patch set uploaded (by BBlack):
Revert "ipsec-for-kafka: limit to kafka1012 cp4011 for testing" Revert "post-merge fixup for 8b9dfe360"

https://gerrit.wikimedia.org/r/259280

Change 259280 merged by BBlack:
Revert "ipsec-for-kafka: limit to kafka1012 cp4011 for testing"

https://gerrit.wikimedia.org/r/259280

Change 259290 had a related patch set uploaded (by BBlack):
kafka::server: allow ipsec traffic

https://gerrit.wikimedia.org/r/259290

Change 259290 merged by BBlack:
kafka::server: allow ipsec traffic

https://gerrit.wikimedia.org/r/259290