Thin out old revision renders
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• GWicke
	Mar 27 2015, 5:10 PM

Description

Before implementing the recent re-render optimizations we stored a lot more re-renders than necessary. We should thin those out to only leave a single / latest entry per revision.

We might also want to implement time-based storage policies, for example to only keep one render per 24-hour interval. This should probably be implemented in the cassandra backend, so that the retention policy can be configured in the schema. This is discussed in T94524: Configurable garbage collection / revision retention policy in table schemas: add 'interval' policy support.

Related Objects
Search...

Status	Assigned	Task
Resolved	Catrope	T90374 Retrieve page HTML directly from RESTbase
Resolved	• GWicke	T93291 Disable Parsoid cache updates for wikis that are migrated to RESTBase
Resolved	• GWicke	T93452 Enable group 1 wikis in RESTBase
Invalid	None	T93751 RFC: Next steps for long-term revision storage -- space needs, storage hierarchies
Resolved	• GWicke	T94196 Thin out old revision renders

Event Timeline

• GWicke created this task.Mar 27 2015, 5:10 PM

• GWicke raised the priority of this task from to High.

• GWicke updated the task description. (Show Details)

• GWicke added a project: RESTBase.

• GWicke added subscribers: Eevans, • mobrovac, • GWicke.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2015, 5:10 PM

• GWicke added a parent task: T93452: Enable group 1 wikis in RESTBase.Mar 27 2015, 5:11 PM

• GWicke mentioned this in T93751: RFC: Next steps for long-term revision storage -- space needs, storage hierarchies.

PR at https://github.com/wikimedia/restbase-mod-table-cassandra/pull/87.

Liuxinyu970226 subscribed.Mar 28 2015, 1:01 AM

• GWicke updated the task description. (Show Details)Mar 28 2015, 3:39 PM

• GWicke set Security to None.

It took a couple of attempts and modifications to the script to actually make it reliable, but four copies of it have now been running since yesterday afternoon with the following result:

Some observations:

it is really common for articles to have > 10 renders per revision
an interesting class of often-rerendered articles are user-created admin dashboards (often User:Something/dashboard)

• GWicke mentioned this in T94422: Consistently use the same render for html2wt processing after an edit.Mar 30 2015, 6:54 PM

Compaction is picking up more of the slack now:

I think we can move ahead with group 1 projects now.

• GWicke mentioned this in T94524: Configurable garbage collection / revision retention policy in table schemas: add 'interval' policy support.Mar 31 2015, 12:59 AM

• GWicke updated the task description. (Show Details)

Another screenshot:

Closing for the manual part.

These scripts have been run about once per day since ticket creation, which in combination with additional optimizations implemented earlier was very effective at limiting the growth rate. The most heavily loaded node now stores 800G. The growth rate is currently only about 1G per day, but the actual rate might actually be higher as there are still a lot of tombstones being processed by compaction. We'll have to wait for a bit longer to establish new growth rates.

Liuxinyu970226 unsubscribed.Apr 7 2015, 5:02 AM

	F107893: pasted_file
	Apr 1 2015, 7:57 PM

	F107212: pasted_file
	Mar 31 2015, 12:05 AM

	F107104: pasted_file
	Mar 30 2015, 6:06 PM

Thin out old revision rendersClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Thin out old revision renders
Closed, ResolvedPublic
Actions

Related Objects
Search...