Page MenuHomePhabricator

Create a script to easily query and remove backups from the media storage backups (primarily to attend T&S deletion requests)
Closed, ResolvedPublic

Description

From time to time we (data persistence) get Trust and Safety requests to delete already production deleted soft-deleted files permanently, for several legal reasons. While the whole architecture was built around having this functionality available, we first focused on automated restoration, and at the moment, hard deletions from backups is not an easy task, it is very manual and time consuming, and requires quite specialized knowledge (SQL queries, database model knowledge, minio structure and distribution, etc.).

Create, similarly to how the restore-media-file script works a couple of interactive scripts to query and delete media files from its backups, easy enough that any sysadmin with basic media storage and media backups knowledge can use it, but safe enough it cannot accidentally cause data loss.

Event Timeline

Change 807931 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] restore_media_file: Split restore-media-file cli into it and a library

https://gerrit.wikimedia.org/r/807931

Change 807931 merged by Jcrespo:

[operations/software/mediabackups@master] restore_media_file: Split restore-media-file cli into it and a library

https://gerrit.wikimedia.org/r/807931

Change 807961 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] mediabackups: Create querying-only cli utility

https://gerrit.wikimedia.org/r/807961

Change 807961 merged by Jcrespo:

[operations/software/mediabackups@master] mediabackups: Create querying-only cli utility

https://gerrit.wikimedia.org/r/807961

jcrespo renamed this task from Create a script to easily query and remove backups from the media storage backups (primarilly to attend T&S deletion requests) to Create a script to easily query and remove backups from the media storage backups (primarily to attend T&S deletion requests).Jun 23 2022, 3:26 PM

Change 808013 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Add new script delete-media-file to delete backed up files

https://gerrit.wikimedia.org/r/808013

Change 808314 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] delete-media-file: Add failsafe to file deletion

https://gerrit.wikimedia.org/r/808314

Change 808013 merged by Jcrespo:

[operations/software/mediabackups@master] Add new script delete-media-file to delete backed up files

https://gerrit.wikimedia.org/r/808013

Change 808314 merged by Jcrespo:

[operations/software/mediabackups@master] delete-media-file: Add failsafe to file deletion

https://gerrit.wikimedia.org/r/808314

Change 809145 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Prepare for 0.1.2 release

https://gerrit.wikimedia.org/r/809145

Change 809152 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] delete-media-backups: Default to dry-mode for deletions

https://gerrit.wikimedia.org/r/809152

Change 809145 merged by Jcrespo:

[operations/software/mediabackups@master] Prepare for 0.1.2 release

https://gerrit.wikimedia.org/r/809145

Change 809152 merged by Jcrespo:

[operations/software/mediabackups@master] delete-media-backups: Default to dry-mode for deletions

https://gerrit.wikimedia.org/r/809152

Change 809588 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] Prepare for 0.1.3 release

https://gerrit.wikimedia.org/r/809588

Change 809589 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] cli: Change logging to log on a different file each

https://gerrit.wikimedia.org/r/809589

Change 810037 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/mediabackups@master] InteractiveQuery: Add additional cli messages after user testing

https://gerrit.wikimedia.org/r/810037

Example run and screenshots: P30677 Thanks to @Marostegui for the feedback!

Change 809588 merged by Jcrespo:

[operations/software/mediabackups@master] Prepare for 0.1.3 release

https://gerrit.wikimedia.org/r/809588

Change 809589 merged by Jcrespo:

[operations/software/mediabackups@master] cli: Change logging to log on a different file each

https://gerrit.wikimedia.org/r/809589

Change 810037 merged by Jcrespo:

[operations/software/mediabackups@master] InteractiveQuery: Add additional cli messages after user testing

https://gerrit.wikimedia.org/r/810037

Considered resolved, documentation can be found at: https://wikitech.wikimedia.org/wiki/Media_storage/Backups#Deleting_files (in addition to the rest of the page).