In T156199#2970308, @Eevans wrote:

I don't have strong feelings about truncating the parsoid tables, except for a concern that it sets an expectation that the OOMs will cease; I don't think there is any reason to believe that either of T153588 or T156155 could have been avoided by truncating the tables. These were the result of accumulated tombstones within a single revision, something we'd continue to be vulnerable to even if we're only retaining the current revisions, (possibly more so given the other proposed changes, latest_hash, TTL renewal, etc).

In case there is any doubt about this, this can be used to reproduce the issue. The resulting heap dump looks identical what we found in T153588:

The conditions for this are a function of the size of the value, the number of renders for a revision, and the efficacy of tombstone GC; It is possible for this to occur on an otherwise empty database with only a single stored revision.

@Eevans, I don't think there was any doubt about this being possible given enough re-renders in a short amount of time. However, this does not tell us much about whether this would matter to us in practice when storing current revisions only, and assuming upper bounds on re-render rates as discussed in T156199#2973062.

In any case, we just discussed this in our 1:1, and you made it clear that you would prefer to not rely on tombstone GC to happen within a bounded time. @Pchelolo started a design discussion for schemas that avoid this at the cost of some complexity, so lets look at those options, and then consider our next steps in Thursday's team meeting.

Changed the table names to be a bit more generic, in line with the cassandra table implementation.

One big area we have not defined yet is how the mapping between high-level table schemas & cassandra schemas would work. For the concrete case we are considering here, the only realistic option I see right now is to keep defining revision as a range key in the table spec, and then modify the Cassandra schema for the data table to omit this key when the latest_hash retention policy is set. The ttl retention policy is fairly easy to handle as well (only using the ttl table). I'm less sure about the other retention policies, or the case where no retention policy is defined at all.

Another question is what we will do about listing support. A lot of the schema changes we are discussing here are making listings impossible. Should we generally remove support for listings from the table spec, reflecting the lowest common denominator?

Edit: Added a section with some thoughts on this in the next steps doc.

Since the If-Modified-Since timestamp has a one second resolution, we can numerically add the revision number to the write time to guard against sub-second race conditions.

We'll probably need to come up with something a little more clever; Revision ID is an unsigned int and simply adding the revision to a microseconds representation of the timestamp will skew the writetime (increasing) into the future (which I don't think we want).

In T156199#3009590, @Eevans wrote:

Since the If-Modified-Since timestamp has a one second resolution, we can numerically add the revision number to the write time to guard against sub-second race conditions.

We'll probably need to come up with something a little more clever; Revision ID is an unsigned int and simply adding the revision to a microseconds representation of the timestamp will skew the writetime (increasing) into the future (which I don't think we want).

We could make sure that whatever we derive from the revision is less than one second's worth of microseconds.

CREATE TABLE ttl (
  "_domain" text,
  time_window bigint,
  title text,
  revision int,
  tid timeuuid,
  html text,
  data_parsoid text,
  section_offsets text,
  PRIMARY KEY (("_domain", time_window, title, revision), tid)
);

What was the reason for using tid as a cluster key here (I mean as opposed to another component of the partition key)? In other words, is there any reason not to use the following instead?

CREATE TABLE ttl (
  "_domain" text,
  time_window bigint,
  title text,
  revision int,
  tid timeuuid,
  html text,
  data_parsoid text,
  section_offsets text,
  PRIMARY KEY (("_domain", time_window, title, revision, tid))
);

What was the reason for using tid as a cluster key here (I mean as opposed to another component of the partition key)? In other words, is there any reason not to use the following instead?

We wouldn't always have a tid in a request. For example for requests for HTML with a revision but without the TID (if one is starting to edit a specific revision of the content)

Not going deeper in reasoning on the requirements that this tickets assume to be true (I'm not sure all of those are justified, but that's another topic) I would say that the "application-level TTLs" options seems the best way to go for a few reasons:

1. It doesn't multiply the write load
2. It doesn't increase significantly the complexity of the schema
3. It should mostly keep the amount of storage we have to do in check
4. Seems simple enough not to trip over race conditions.

I would also add that the requirement stated on the mediawiki.org page

"Storage of arbitrarily old revisions (on request), for a TTL period (at least), from the time of the request" is, to my understanding, only valid in the context of Visual Editor. If that's the case, I would strongly suggest we save arbitrary old revisions only if the request has some 'secret' hashing in the query that signals it's coming from VE.

Else, it would only take a pretty persistent bot to render enough old revisions to both cripple storage performance and fill it up.

Option: Separate TTL table with rage deletes on latest content

Rage deletes indeed. :)

Thanks for adding Option 2 (TTL table, range deletes on latest, no timeline)! One advantage of this option that @Pchelolo & I were pretty excited about is that this should generalize well to arbitrary schemas, which will make it a lot easier to implement the table storage spec.

Alternative (slightly tweaked) update sequence for option 2:

• read previously-latest render

BATCH:

• ttl table: write previously latest render to ttl table
• latest table: insert using TID derived timestamp
• latest table: range delete from latest table
  • smaller revision numbers
  • same revision, lower tid
  • (generally, any key range that is lower, at each range index level)
• ttl table: insert using TID

Considering all three options it seems option 1 > option 2 > option 3.
In particular in option 2 though I'm not sure the pros outweight the cons as it seems more complex and higher latency.

Further, I couldn't find any estimation of latency requirements for current vs historical storage and estimated size of "current" storage. Also how revisions requested by VE (assuming it is the most latency-sensitive rb client) are distributed, IOW how they would fall into current vs historical buckets

In T156199#3307062, @fgiunchedi wrote:

Further, I couldn't find any estimation of latency requirements for current vs historical storage and estimated size of "current" storage.

Good questions.

For latency, I think the short (unsatisfying?) answer is that for current storage it should be at least as good as what we have now (https://grafana.wikimedia.org/dashboard/db/restbase?fpanelId=11&fullscreen&orgId=1), while archive storage can be somewhat more latent. But you're right, we should put real numbers to this.

For storage size of current data, I'm not sure how to do much better than a SWAG. Average content size x number-of-titles x overhead(s) x 1.5 (per storage group). Or maybe we can infer it from the dev env. I'll try to work on this.

Also how revisions requested by VE (assuming it is the most latency-sensitive rb client) are distributed, IOW how they would fall into current vs historical buckets

Right; I'm curious about this as well. It feels like something that is pretty exceptional. If so, maybe it should be handled that way.

We got a good idea of size & expected latency right after we set up the RB cluster, and filled it with current revisions. Size for current revisions (all domains) was ~400G compressed total, access latency (from memory) < 20ms, with p50 <5ms.

In T156199#3307062, @fgiunchedi wrote:

Considering all three options it seems option 1 > option 2 > option 3.
In particular in option 2 though I'm not sure the pros outweight the cons as it seems more complex and higher latency.

Further, I couldn't find any estimation of latency requirements for current vs historical storage and estimated size of "current" storage. Also how revisions requested by VE (assuming it is the most latency-sensitive rb client) are distributed, IOW how they would fall into current vs historical buckets

Added some latency requirements to the document