Page MenuHomePhabricator
Create Task
Maniphest T189845

Ensure discovery.query_clicks_* data is purged per privacy policy
Closed, ResolvedPublic
Actions

Description

The click data is unaggregated and includes PII, it needs to be deleted after 90 days to match our privacy policy. The hourly table is maintained by hive and typically has only a day of data. The daily table is, currently, pruned by manually calling a script and is done intermittently.

Task is to make patches necessary so the script is called automatically on a daily basis. We might as well purge both the hourly and daily tables from the script just incase our oozie pipeline fails to delete some hours from hive (has happened before).

Details

SubjectRepoBranchLines +/-
Drop query_clicks partitions after 90 daysoperations/puppetproduction+14 -0
Support hourly or daily partition droppinganalytics/refinerymaster+42 -29
Customize query in gerrit

Event Timeline

EBernhardson created this task.Mar 15 2018, 11:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 15 2018, 11:55 PM
EBernhardson moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.Mar 15 2018, 11:55 PM
gerritbot subscribed.Mar 15 2018, 11:56 PM

Change 419949 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[analytics/refinery@master] Support hourly or daily partition dropping

https://gerrit.wikimedia.org/r/419949

gerritbot added a project: Patch-For-Review.Mar 15 2018, 11:56 PM
gerritbot added a comment.Mar 16 2018, 12:03 AM

Change 419949 abandoned by EBernhardson:
Support hourly or daily partition dropping

Reason:
after some testing it looks like refinery-drop-hive-partitions will work to do exactly what this patch is trying to change the script to do.

https://gerrit.wikimedia.org/r/419949

gerritbot added a comment.Mar 16 2018, 12:30 AM

Change 419954 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Drop query_clicks partitions after 90 days

https://gerrit.wikimedia.org/r/419954

EBernhardson moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Mar 16 2018, 12:35 AM
• EBjune moved this task from Needs review to Waiting on the Discovery-Search (Current work) board.Apr 17 2018, 5:35 PM
EBernhardson added a comment.Apr 17 2018, 5:43 PM

Once the latest refinery version is deployed we can merge the cron that regularly drops old data.

debt triaged this task as Medium priority.May 1 2018, 5:27 PM
EBernhardson added a comment.Jun 7 2018, 4:08 PM

Checked and the code was deployed, but there was a new bug in refinery preventing the partition drop from working. https://gerrit.wikimedia.org/r/438034 has been merged and once deployed we should be ready to go.

EBernhardson moved this task from Waiting to Needs review on the Discovery-Search (Current work) board.Jul 30 2018, 10:46 PM
EBernhardson added a subscriber: Gehel.

@Gehel It looks like the analytics deploy has happened, i tested the script today and everything seems to work great. The linked gerrit patch for puppet (https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/419954/) should now be ready for deployment

gerritbot added a comment.Jul 31 2018, 3:54 PM

Change 419954 merged by Gehel:
[operations/puppet@production] Drop query_clicks partitions after 90 days

https://gerrit.wikimedia.org/r/419954

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Jul 31 2018, 5:00 PM
debt closed this task as Resolved.Jul 31 2018, 5:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL