Most of the request timeouts in logstash are from user dashboards that are constantly re-rendered by update jobs. They normally transclude fast-changing pages like https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion, which triggers the re-renders.

The query looks like this:

select "key","rev","tid","latestTid","value","content-type","content-sha256","content-location","tags","_domain","_del" from "local_group_wikipedia_T_parsoid_html"."data" where "key" = ? AND "rev" = ? AND "_domain" = ?

It doesn't contain a limit, which is odd. Originally I thought it came from background updates, but the absence of the limit makes me less sure about this. The relevant accesses in key_rev_bucket seem to specify a limit as well.

Edit: I think these queries come from https://github.com/wikimedia/restbase/blob/master/mods/key_rev_value.js#L200-L205. The limit: 1 there is being converted to fetchSize: 1 in getRaw. dbu.buildGetQuery then ignores the query.limit parameter, which results in it being omitted from the query. A PR to clarify this in dbutils is at https://github.com/wikimedia/restbase-mod-table-cassandra/pull/161.

Further investigation shows that requests for some of these articles with a lot of expired renders reliably fail, despite limit 1 or fetchSize: 1. The likely cause for this seems to be Cassandra reading all tombstones in fully expired SSTable levels, which would also cause higher-than-usual memory pressure and read latencies.

Given a fixed re-render rate, the factors that will affect the number of tombstones kept around per partition are:

• gc_grace interval, currently set to seven days.
• Timeliness of compaction after gc_grace elapses. We are using leveled compaction, which generally ensures fairly timely compaction at the cost of more IO.
  • Possibly, leverage date tiered compaction?

The most obvious solution would be to reduce gc_grace. The downside could be deleted data coming back when a node re-joins after more than max_hint_window_in_ms (default 3 hours). While a node being out or slow for a couple of hours does happen in practice, it seems less likely that a node would be out for more than ~12 hours. I do thus wonder if we could set both max_hint_window_in_ms and gc_grace to ~12 hours, along with the rule to wipe nodes that are down for longer than this time. During periods of general trouble likely to last > 12 hours we could increase gc_grace dynamically, as needed. Relying more on hints should be especially viable with new hint storage in 3.0, but there are also some optimizations since 2.1.1 that should make > 3 hours supportable at our relatively modest update rates. This discussion has at least one user reporting a use of at least 6 hour hint windows, but cautioning against going all-out with > 24 hours in Cassandra 2.1.

After looking some more into date tiered compaction, it actually appears to be a very good fit for key_rev_value and key_rev buckets, and more generally anything that is versioned and predominantly produced in timestamp order. Even with out-of order production (backfilling) of old revisions, access to new revisions should be reliably fast thanks to the non-overlapping timestamp ranges for new revisions. The non-overlapping property avoids the need to read loads of tombstones in lower levels, along with the associated memory pressure / OOM risk and timeouts.

To evaluate performance for our use case in practice, I have switched the parsoid html and data-parsoid buckets to the DateTieredCompactionStrategy. I have also started a forced re-render of all enwiki articles (with no-cache), to simulate a typical re-render workload.

I created a dashboard for staging: https://grafana-admin.wikimedia.org/dashboard-import/restbase-staging-cassandra

The re-compaction is now mostly finished.

First results:

Read latency