Page MenuHomePhabricator
Create Task
Maniphest T168614

Add a job that regularly deletes druid webrequest deep-stored data
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Currently, druid keeps 7 days of sampled webrequests in the hitorical nodes.
However data is not deleted from deep-storage, putting us in breach of data rentention policy.
We should have a job that deletes segments older than 60 days. Keeping that many data is just in case some data-emergency occurs, we'll be able to relaod it easily.
Hoiw to delete deep-storage segment: Second part of https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid#Delete_a_data_set_from_deep_storage

  • Script deleting data
  • Puppetization of script on webrequest datasource

Details

SubjectRepoBranchLines +/-
role::analytics_cluster::refinery::job::data_drop: drop old druid dataoperations/puppetproduction+17 -4
Upgrade script dropping druid deep-storage dataanalytics/refinerymaster+160 -77
Add script deleting druid deep storage dataanalytics/refinerymaster+127 -0
Customize query in gerrit

Event Timeline

JAllemandou created this task.Jun 22 2017, 10:28 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 22 2017, 10:28 AM
• Nuria edited projects, added Analytics-Kanban; removed Analytics.Jun 26 2017, 3:50 PM
JAllemandou claimed this task.Jun 27 2017, 10:09 AM
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot subscribed.Jun 27 2017, 11:35 AM

Change 361651 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Add script deleting druid deep storage data

https://gerrit.wikimedia.org/r/361651

gerritbot added a project: Patch-For-Review.Jun 27 2017, 11:35 AM
JAllemandou added a comment.Jun 27 2017, 11:37 AM

First version of a python script deleting data (tested, obviously).
After CR and when deployed, let's not forget to automate it for webrequest datasource using a puppet cron.

JAllemandou updated the task description. (Show Details)Jun 27 2017, 11:38 AM
JAllemandou moved this task from In Progress to In Code Review on the Analytics-Kanban board.Jun 27 2017, 3:03 PM
JAllemandou moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Jun 28 2017, 3:04 PM
gerritbot added a comment.Jun 28 2017, 3:29 PM

Change 361651 merged by Joal:
[analytics/refinery@master] Add script deleting druid deep storage data

https://gerrit.wikimedia.org/r/361651

• Nuria subscribed.Jun 28 2017, 4:58 PM

We need to add corresponding puppet code to execute this script as a cron

JAllemandou moved this task from Ready to Deploy to In Code Review on the Analytics-Kanban board.Jun 29 2017, 8:07 AM
gerritbot added a comment.Jun 29 2017, 8:08 AM

Change 362148 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Add cron job dropping webrequest from druid

https://gerrit.wikimedia.org/r/362148

gerritbot added a comment.Jun 30 2017, 12:23 PM

Change 362396 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Upgrade script dropping druid deep-storage data

https://gerrit.wikimedia.org/r/362396

gerritbot added a comment.Jun 30 2017, 4:50 PM

Change 362396 merged by Joal:
[analytics/refinery@master] Upgrade script dropping druid deep-storage data

https://gerrit.wikimedia.org/r/362396

gerritbot added a comment.Jul 4 2017, 11:10 AM

Change 362148 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::refinery::job::data_drop: drop old druid data

https://gerrit.wikimedia.org/r/362148

JAllemandou moved this task from In Code Review to Done on the Analytics-Kanban board.Jul 5 2017, 12:16 PM
JAllemandou set the point value for this task to 5.Jul 6 2017, 8:32 AM
• Nuria closed this task as Resolved.Jul 10 2017, 3:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL