Page MenuHomePhabricator

Estimate storage capacity needed for storing all HTML revisions
Closed, ResolvedPublic

Description

Eventually we'd like to store all article revisions as HTML. This task tries to establish a rough guess for the storage we'd need for this.

  • After ~1 1/2 months of operation, all HTML revisions rendered across all wikipedias take up about 6T of space in the cluster. This does not include non-article pages that were not edited or otherwise re-rendered in this period. The size of articles only right after cluster setup was about 1.5T (all three-way replicated).
  • On enwiki, the ratio of pages to articles in ns0 is ~36 million to ~5 million.
  • In the large wikis, the mean number of revisions is around 20 revisions / article (enwiki: 21, dewiki: 28, frwiki: 17). Smaller wikis tend to have less edits per article.

Assuming a mean of 20 revisions per page and three-way replication, this means that:

  • we'd need at least ~20 * 1.5T = 30T of storage for articles (ns0) only, and
  • we'd need at least 20 * 6T = ~120T of storage for all pages.

The assumption of 20 revisions / page and uniform page size is probably conservative, so there's a non-zero chance that these numbers could actually work out in real life.

Event Timeline

GWicke raised the priority of this task from to Medium.
GWicke updated the task description. (Show Details)
GWicke added a project: RESTBase.
GWicke claimed this task.
GWicke moved this task from In progress to Done on the RESTBase board.