Page MenuHomePhabricator

Set up automatic deletion for netflow datasource in Druid
Closed, ResolvedPublic

Description

After the latest additions to it, the netflow data set contains a pair of privacy-sensitive fields.
This means we should either delete or sanitize this data set after 90 days after collection.
This task is to implement such deletion/sanitization for the netflow data in Druid (Turnilo).
This task won't affect the netflow data in Hive.

Initially, I see 2 options:

  1. Entirely delete the data in Druid that is older than 90 days (or 3 months). This could be easily done just adding a configuration to the Druid coordinator.
  2. Reload the netflow data to Druid at 90 days after collection, but with the privacy sensitive fields nullified (or just absent). This is theoretically quite simple as well, we'd have to add another timer to puppet with less fields. But it needs a bit of research and testing, not sure what will be the behavior of Druid if we override existing data with new data that has another schema (some fields missing). Also, if we chose to go with this solution, we'd have to also delete the data from Druid as described in option 1) after a given period of time, say 1 year (2 years?).

Event Timeline

Ottomata triaged this task as High priority.
Ottomata added a project: Analytics-Kanban.
Ottomata moved this task from Incoming to Operational Excellence on the Analytics board.
mforns renamed this task from Set up a deletion timer for netflow data set to Set up automatic deletion for netflow datasource in Druid.Aug 27 2019, 3:34 PM
mforns updated the task description. (Show Details)
mforns added a subscriber: ayounsi.

Hi @ayounsi!

Please, review this task and let us know how long would you like to keep the netflow data in Druid/Turnilo.
Or put in another way, how interesting is it to you, to have the netflow data accessible in Druid/Turnilo for a long time?
This task is not about the netflow data in Hive/HDFS, I'll create another one for that :-)

Thanks a lot!

It would be great to have an aggregated/sanitized set of the data for as long as possible.

In addition to ip_src and ip_dst it's fine to remove port_src port_dst tcp_flags ip_proto. Leaving AS information, direction, and future country information. Hopefully that helps keeping the DB size reasonable.
Similarly, aggregating the data with a 5min or more granularity is fine.

@ayounsi
Great, thanks.
I'm not sure if we can change the granularity of the data within a single data set, say have the latest 3 months be minutely, and the rest be 5-minutely. I assume not.
But I will start testing how Druid/Turnilo behave when overriding existing data with new data that does not contain the fields you mentioned.

Change 535924 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] analytics::refinery::job::druid_load: Add sanitization for netflow

https://gerrit.wikimedia.org/r/535924

That patch should do the trick,
but we should wait about 2 months before merging.
Netflow data from 90 days ago still has the old schema and would produce useless and confusing data.
In 60 days, we can merge this and will hopefully work.

Yes, @Nuria, the data starts 17th of August, so we can merge end of next week. Or on Monday the 18th? Better chance of having an ops person to merge the change.

Change 535924 merged by Elukey:
[operations/puppet@production] analytics::refinery::job::druid_load: Add sanitization for netflow

https://gerrit.wikimedia.org/r/535924

Change 552082 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] analytics::refinery::job::druid_load: fix neflow sanitization

https://gerrit.wikimedia.org/r/552082

Change 552082 merged by Elukey:
[operations/puppet@production] analytics::refinery::job::druid_load: fix neflow sanitization

https://gerrit.wikimedia.org/r/552082