Of the alternatives discussed for managing retention of current revision storage, [[ https://www.mediawiki.org/wiki/RESTBase/StorageDesign#Retention_policies_using_application-level_TTLs | the use of range deletes ]] seems most promising. This issue will track the prototyping and testing of this design.
== Outline ==
[x] Deploy [[ https://github.com/wikimedia/change-propagation | change-propagation ]] to the dev environment, configured to run in `test_mode`, and with sampling enabled
[x] Implement a time-line storage endpoint in RESTBase using a key-value table
[x] Configure change-propagation to send updates to the time-line for new revisions
[] Implement alternative RESTBase retention policy incorporating range deletes
[] Collect data, test, etc
== Current Status (2017-05-17T13-05:00) ==
- An instance of change-prop is running on restbase-dev1003 in `test_mode`, with 15% sampling
- An instance of change-prop is running on restbase-test2003 in `test_mode`, sampling at 15%, (sends to restbase-dev100x, processes re-renders)
- An [[ https://github.com/wikimedia/restbase/compare/wikimedia:master...eevans:storage_proto | experimental branch ]] of RESTBase has been deployed to the dev env
- Adds support for a RESTBase k/v-based time-line store (which change-prop is updating)
- Changes Parsoid tables to use `latest_hash` retention policy
- Uses an [[ https://github.com/wikimedia/restbase-mod-table-cassandra/compare/master...eevans:storage_proto | experimental branch of restbase-mod-table-cassandra ]] that implements a 1 hour TTL on revisions using range deletes for `latest_hash`
Next-up: Experimental support for range delete-based retention of renders