Page MenuHomePhabricator

Refactor druid data deletion script
Open, HighPublic

Description

The script deleting druid-public datasources fails silently if errors accur. Also, it doesn't handle redirects correctly (for deletes DELETE actions). Finally, it uses a to-be-deprecated deletion method for segments, and generates one action per segment (we have experienced a broker issue due to server high load after deletion of a datasource) .
This task is about refactoring the existing DruidDataDropper from therefinery-drop-druid-deep-storage-data script into the DruidUtils object.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2019, 1:59 PM
elukey added a subscriber: elukey.Apr 4 2019, 2:01 PM
elukey added a comment.Apr 4 2019, 2:06 PM

self.delete in utils.py should simply, in my opinion, return http exceptions if any so we can get them in the drop script and sys.exit(1)

fdans moved this task from Operational Excellence to Ops Week on the Analytics board.
fdans triaged this task as High priority.
JAllemandou claimed this task.
JAllemandou renamed this task from Fix druid-public drop-snapshot script to Refactor druid data deletion script.Apr 5 2019, 4:46 PM
JAllemandou updated the task description. (Show Details)
JAllemandou updated the task description. (Show Details)Apr 5 2019, 4:48 PM

Change 502469 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Refactor python util.py into smaller files

https://gerrit.wikimedia.org/r/502469

Change 502469 merged by Milimetric:
[analytics/refinery@master] Refactor python util.py into smaller files

https://gerrit.wikimedia.org/r/502469

Ping @fdans - The patch is not listed here (don't know why), but still needs your review please: https://gerrit.wikimedia.org/r/c/analytics/refinery/+/502858