Page MenuHomePhabricator

Refactor druid data deletion script
Closed, ResolvedPublic3 Estimated Story Points


The script deleting druid-public datasources fails silently if errors accur. Also, it doesn't handle redirects correctly (for deletes DELETE actions). Finally, it uses a to-be-deprecated deletion method for segments, and generates one action per segment (we have experienced a broker issue due to server high load after deletion of a datasource) .
This task is about refactoring the existing DruidDataDropper from therefinery-drop-druid-deep-storage-data script into the DruidUtils object.

Event Timeline

self.delete in should simply, in my opinion, return http exceptions if any so we can get them in the drop script and sys.exit(1)

fdans moved this task from Incoming to Operational Excellence on the Analytics board.
fdans moved this task from Operational Excellence to Ops Week on the Analytics board.
JAllemandou renamed this task from Fix druid-public drop-snapshot script to Refactor druid data deletion script.Apr 5 2019, 4:46 PM
JAllemandou updated the task description. (Show Details)

Change 502469 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Refactor python into smaller files

Change 502469 merged by Milimetric:
[analytics/refinery@master] Refactor python into smaller files

Ping @fdans - The patch is not listed here (don't know why), but still needs your review please:

Nuria set the point value for this task to 3.Jun 10 2019, 5:18 PM