Page MenuHomePhabricator

Research storage solutions for media backups
Closed, ResolvedPublic

Description

As a person responsible for the WMF backups I want to select the optimal storage solution for backing up media files.

Requirements (as defined in T262669):

  • Have a "logical" approach to files (store individual files, not filesystem copies- application agnostic)
  • Store also metadata about the files (name, hash, current status: soft-deleted/archived/wiped)
  • Scale to the sizes WMF hosts
  • Be able to quickly backup and restore a large amount of files
  • Support encryption on the wire and at rest
  • Files themselves will be insert-only, but had to be able to be deleted and dynamically altered its state to reflect production changes

Acceptance criteria:

  • A set of storage options is selected for evaluation
  • Testing methodology is defined
  • Evaluation and tests are performed
  • A list of pros and cons for each storage option is created and published for review

Event Timeline

herron triaged this task as Medium priority.Sep 30 2020, 5:17 PM
LSobanski mentioned this in Unknown Object (Task).Oct 6 2020, 8:59 AM
LSobanski mentioned this in Unknown Object (Task).
jcrespo claimed this task.

Research was done and reflected at private document https://docs.google.com/document/d/1kmaDIrae4HsE1w8x7xRxM_Hv2-gO0VNTvif240aQW4A waiting for ok from higher-ups to publish results on Wikitech.

jcrespo raised the priority of this task from Medium to High.

Research (analysis) and Design finished for now, we are now in implementation phase: T276442 and T276445.

Documentation to be released soon.