Change Details

RESTBase is currently storing HTML and data-parsoid for one or more renders of each revision. Compression ratios are currently: - 14.4% for data-parsoid - 16.9% for html At the current template update rates (~50/s) we are seeing a compressed storage growth on the order of 40G/day. This means that the currently provisioned 2.5T per node in a six-node cluster with three-way replication will only last for ~80 more days. There are several options we can explore: - Only store a render if it actually changed. Many template edits don't actually change the rendering. Unfortunately, Parsoid HTML contains fairly random about attributes that make equality testing difficult: T93715 - Only store one render per revision. This would likely cut the storage requirements to ~1/3 - 1/4. - Implement a better compression scheme like LZMA: T93496 - Implement a storage hierarchy in which only recent revisions are stored in the first-level cluster while older ones are stored on an archive cluster. This could be interesting for geo-located RB installations as well, where we don't need lowest-latency access to historical revision data.

RESTBase is currently storing HTML and data-parsoid for one or more renders of each revision. Compression ratios are currently: - 14.4% for data-parsoid - 16.9% for html At the current template update rates (~50/s) we are seeing a compressed storage growth on the order of 40G/day. This means that the currently provisioned 2.5T per node in a six-node cluster with three-way replication will only last for ~80 more days. There are several options we can explore: - Only store a render if it actually changed. Many template edits don't actually change the rendering. Unfortunately, Parsoid HTML contains fairly random about attributes that make equality testing difficult: T93715 - Only store one render per revision. This would likely cut the storage requirements to ~1/3 - 1/4. - Implement a large-window compression scheme like LZMA: T93496 - Implement a storage hierarchy in which only recent revisions are stored in the first-level cluster while older ones are stored on an archive cluster. This could be interesting for geo-located RB installations as well, where we don't need lowest-latency access to historical revision data.

RESTBase is currently storing HTML and data-parsoid for one or more renders of each revision. Compression ratios are currently: - 14.4% for data-parsoid - 16.9% for html At the current template update rates (~50/s) we are seeing a compressed storage growth on the order of 40G/day. This means that the currently provisioned 2.5T per node in a six-node cluster with three-way replication will only last for ~80 more days. There are several options we can explore: - Only store a render if it actually changed. Many template edits don't actually change the rendering. Unfortunately, Parsoid HTML contains fairly random about attributes that make equality testing difficult: T93715 - Only store one render per revision. This would likely cut the storage requirements to ~1/3 - 1/4. - Implement a betterlarge-window compression scheme like LZMA: T93496 - Implement a storage hierarchy in which only recent revisions are stored in the first-level cluster while older ones are stored on an archive cluster. This could be interesting for geo-located RB installations as well, where we don't need lowest-latency access to historical revision data.