We need to set up an automatic purging system in EventLogging databases:
- By default, all records older than 90 days should be immediately deleted for all tables. This should automatically include tables created in the future.
- The system must support some kind of white-listing (we still need to decide what and how). If a table is white-listed, keep the white-listed fields and auto-purge the others normally.
- clientIp and userAgent fields (eventCapsule) can not be white-listed, they must be always deleted after 90 days.
We need to decide how to implement this in both master and analytics-storage replica.
The schema+fields whitelist needs still to be created from the EL audit spreadsheed, column N:
https://docs.google.com/spreadsheets/d/1mQtbsbGHLbGsHeNaYFCdg6X4K1VSxjUStBpx4GmVbhk/edit#gid=1610723354