Change Details

Currently, druid keeps 7 days of sampled webrequests in the hitorical nodes. However data is not deleted from deep-storage, putting us in breach of data rentention policy. We should have a job that deletes segments older than 60 days. Keeping that many data is just in case some data-emergency occurs, we'll be able to relaod it easily. Hoiw to delete deep-storage segment: Second part of https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid#Delete_a_data_set_from_deep_storage

Currently, druid keeps 7 days of sampled webrequests in the hitorical nodes. However data is not deleted from deep-storage, putting us in breach of data rentention policy. We should have a job that deletes segments older than 60 days. Keeping that many data is just in case some data-emergency occurs, we'll be able to relaod it easily. Hoiw to delete deep-storage segment: Second part of https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid#Delete_a_data_set_from_deep_storage [X] Script deleting data [] Puppetization of script on webrequest datasource

Currently, druid keeps 7 days of sampled webrequests in the hitorical nodes. However data is not deleted from deep-storage, putting us in breach of data rentention policy. We should have a job that deletes segments older than 60 days. Keeping that many data is just in case some data-emergency occurs, we'll be able to relaod it easily. Hoiw to delete deep-storage segment: Second part of https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid#Delete_a_data_set_from_deep_storage [X] Script deleting data [] Puppetization of script on webrequest datasource