Page MenuHomePhabricator

Refactor druid data deletion script
Closed, ResolvedPublic3 Story Points

Description

The script deleting druid-public datasources fails silently if errors accur. Also, it doesn't handle redirects correctly (for deletes DELETE actions). Finally, it uses a to-be-deprecated deletion method for segments, and generates one action per segment (we have experienced a broker issue due to server high load after deletion of a datasource) .
This task is about refactoring the existing DruidDataDropper from therefinery-drop-druid-deep-storage-data script into the DruidUtils object.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2019, 1:59 PM
elukey added a subscriber: elukey.Apr 4 2019, 2:01 PM
elukey added a comment.Apr 4 2019, 2:06 PM

self.delete in utils.py should simply, in my opinion, return http exceptions if any so we can get them in the drop script and sys.exit(1)

fdans triaged this task as High priority.Apr 4 2019, 4:51 PM
fdans moved this task from Incoming to Operational Excellence on the Analytics board.
fdans moved this task from Operational Excellence to Ops Week on the Analytics board.
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
JAllemandou renamed this task from Fix druid-public drop-snapshot script to Refactor druid data deletion script.Apr 5 2019, 4:46 PM
JAllemandou updated the task description. (Show Details)
JAllemandou updated the task description. (Show Details)Apr 5 2019, 4:48 PM

Change 502469 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Refactor python util.py into smaller files

https://gerrit.wikimedia.org/r/502469

Change 502469 merged by Milimetric:
[analytics/refinery@master] Refactor python util.py into smaller files

https://gerrit.wikimedia.org/r/502469

Ping @fdans - The patch is not listed here (don't know why), but still needs your review please: https://gerrit.wikimedia.org/r/c/analytics/refinery/+/502858

mforns moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Jun 6 2019, 2:31 PM
Nuria set the point value for this task to 3.Jun 10 2019, 5:18 PM
Nuria closed this task as Resolved.Mon, Jun 17, 7:55 PM