Netflow is currently pushing data from all datacenters to Kafka-Jumbo without TLS. Since pmacct runs in every DC now and is able (via librkafka) to use TLS towards Kafka, I propose the following:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | odimitrijevic | T257554 Netflow data pipeline | |||
| Resolved | mforns | T231339 Set up automatic deletion/snitization for netflow data set in Hive | |||
| Duplicate | None | T245287 Setup refinment/sanitization on netflow data similar to how it happens for other event-based data | |||
| Open | None | T248865 Automate ingestion of netflow event stream | |||
| Restricted Task | |||||
| Stalled | None | T250146 Add Authentication/Encryption to Kafka Jumbo's clients | |||
| Resolved | elukey | T248980 Move netflow to TLS encryption/authentication via librdkafka |
We could start with TLS encryption only, with:
security.protocol=SSL ssl.ca.location=/etc/ssl/certs/Puppet_Internal_CA.pem ssl.cipher.suites=ECDHE-ECDSA-AES256-GCM-SHA384 ssl.curves.list=P-256 ssl.sigalgs.list=ECDSA+SHA256
Mentioned in SAL (#wikimedia-operations) [2020-04-01T12:17:31Z] <XioNoX> restart nfacct on netflow4001 for kafka tls tests - T248980
Change 585223 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Enable TLS encryption to Kafka Jumbo for netflow4001
Change 585223 merged by Elukey:
[operations/puppet@production] Enable TLS encryption to Kafka Jumbo for netflow4001
Change 585234 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::pmacct: use kafka TLS connection string when needed
Change 585234 merged by Elukey:
[operations/puppet@production] profile::pmacct: use kafka TLS connection string when needed
Change 585255 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Enable TLS encryption to Kafka Jumbo for all pmacct instances
Change 585255 merged by Elukey:
[operations/puppet@production] Enable TLS encryption to Kafka Jumbo for all pmacct instances
For the moment I am happy with TLS encryption only, since we'll probably move to kerberos authentication soon and it doesn't make much sense to do TLS client auth first.