After the latest additions to it, the netflow data set contains a pair of privacy-sensitive fields.
This means we should either delete or sanitize this data set after 90 days after collection.
This task is to implement such deletion/sanitization for the netflow data in Druid (Turnilo).
This task won't affect the netflow data in Hive.
Initially, I see 2 options:
1) Entirely delete the data in Druid that is older than 90 days (or 3 months). This could be easily done just adding a configuration to the Druid coordinator.
2) Reload the netflow data to Druid at 90 days after collection, but with the privacy sensitive fields nullified (or just absent). This is theoretically quite simple as well, we'd have to add another timer to puppet with less fields. But it needs a bit of research and testing, not sure what will be the behavior of Druid if we override existing data with new data that has another schema (some fields missing). Also, if we chose to go with this solution, we'd have to also delete the data from Druid as described in option 1) after a given period of time, say 1 year (2 years?).