Dev cluster is {{done}}; Resolving

The production cluster has finished.

In T204752#4638573, @Pchelolo wrote:

hm... on restbase1007:

c-cqlsh a -e "describe keyspaces;" | grep restr
"commons_T_page__restrictions"

And the dev cluster was forgotten.

Manually dropped others_T_page__restrictions in deployment-prep.

In T197018#4631593, @Eevans wrote:

Upstream issue opened as: CASSANDRA-14797

Upstream issue opened as: CASSANDRA-14797

Confirmed; There are no reads/writes occurring on these keyspaces.

Here it is again after a day; This is definitely something, though not enough to be a game-changer (granted this is a pretty simplistic test).

Here is 1007 after an hour (along with 1016 which is Intel-equipped, for comparison). It's not worse. :)

Here is restbase1008 (first pass) using --verbose

First pass on restbase1007 running fstrim --all took several seconds to complete (a noteworthy delay, 10 seconds?) and exited 0. When run on restbase1011, the command completed almost instantaneously, making me somewhat suspicious. I added a second iteration to each host adding --verbose, restbase1007 outputs bytes trimmed for each mount point, restbase1011 does not (it outputs nothing).

In T89584#4582113, @faidon wrote:

As I mentioned above in my second-to-last update, they are blacklisted for queued TRIM which is suboptimal of course. However, the data corruption issues with synchronous TRIM have been long resolved ­-- they were already back in 2016, they certainly seem to be in the kernels we're running with now.

I think synchronous TRIM would work and will probably make a huge difference. Mounting with discard will have that effect, but the problem with synchronous TRIM (AFAIK) is the potential for stalls on every delete, while you wait for the SSD to actually trim the data. Given our current state of performance, I doubt it can get worse, though :)

A perhaps better alternative would be a periodic (e.g. every hour via cron) invocation of fstrim --all.

As a next step, I'd recommend to just run that (fstrim --all) in one of the systems just once, and compare the performance before/after, either in a production system with real load, or in a system being benchmarked. That very quick test would give us an indication of what kind of benefit we can expect and whether it makes sense to investigate this further.

As an update: These devices are still blacklisted AFAICT.

In T204024#4576751, @daniel wrote:

This use case seems similar to caching parsoid HTML, which is done in RESTbase and backed by Cassandra. It's similar, because it's re-generated upon edit, and accessed from clients upon view, via an API. It's also similar in that losing this data is not absolutely critical, as it can be regenerated, but having to re-generate all of it may cause a problematic spike in load on application servers (and databases and the query service).

However, in contrast to the parsoid use case, information does not need to be stored for old revisions.

As to the model: the wikidata folks will have the details, but as far as I'm aware, it's a JSON blob for each Wikidata entity (items, properties, etc). Granularity could be increased to per-statement blobs.

Puring is, as far as I known, currently only done per edit of the subject. However, use cases for bulk purges exist (in particular, when constraints definitions change), but they are just ignored at the moment, as far as I know. I could be wrong about that, though.

OK, so the dataset size and query rate are quite low and expected to remain so, and the latency and availability expectations aren't particularly onerous. Pretty much anything that can durably store structured data can accommodate these requirements.

In T203039#4549477, @bmansurov wrote:

Data for 50 language pairs is ready. Services, please let me know what the next steps are.

In T203039#4543773, @bmansurov wrote:

@Eevans

[ ... ]

And speaking of understanding the problem: In case I missed it, what is it that makes us want to use Cassandra for this? Other than being able to answer the queries cited, what are the properties we need? For example: Level of fault tolerance, latency, a need for master-master replication, data set size/throughput, etc.

MySQL was considered during the development of the API end points here. I'd be happy to go the MySQL route if we had some way to access MySQL from the service. In this patch @mobrovac mentioned that we can't do so yet. That's why we want to use Cassandra for this. Since we don't know about the potential API usage (except for the current usage in ContentTranslation), I wasn't thinking too much about the features you mentioned.

In T203039#4542081, @bmansurov wrote:

@Pchelolo

[ ... ]

Which queries will you perform on the data? Would you need to fetch the numbers for all pairs in a wikidata item, or a single pair only?

We currently have two use cases for this data (we may have more in the future):

1. For a given list of Wikidata IDs we want to select one or more language pairs that have the same target language. If a language pair is 'en-ko', then the source language is 'en', and the target language is 'ko'. We then want to combine these results into a list and sort Wikidata IDs by the prediction score ('value' in your schema). This use case is mainly derived from the current Content Translation use case.

From the SRE/CP Scalability meeting:

Just for posterity sake: I don't know why the log would have been corrupted like this (almost certainly a bug), but the commitlog only exists to append incoming writes until what was buffered in memory can be flushed to storage. Since AQS replicates data 3 ways, only appends new values, and reads at quorum there is zero risk of losing data by deleting a committlog segment in a situation like this.

LGTM

the system is back up but I don't intend to put it back into rotation until hardware is inspected.

In T201103#4484026, @mobrovac wrote:

In T201103#4482485, @Pchelolo wrote:

A little bit more data: according to webrequest logs mobile-sections-lead was requested 4.808.026 times on 2018/08/03. According to RESTBase graphs, on the same day the average rate of requests reaching RESTbase was 15/s, which gives us 1.296.000 Varnish cache-misses giving the hit ratio of 0.73

I think this data point works in favour of keeping the pre-generation going, since RESTBase obviously alleviates much of the load from MCS and allows for a quick delivery of content to clients (which is RB's base benefit).

Here is an attempt at summarizing the discussion so far (please chime in if any of this is wrong):

In T201103#4484026, @mobrovac wrote:

[ ... ]

• the ratio of mobile vs desktop usage is constantly growing, so what today constitutes 64k requests can easily become several hundred thousands of requests in the near future

Here is a recent histogram of ages for wikipedia_T_mobile__ng_remaining:

Theses are probably queries to system_auth.

If I am reading https://grafana.wikimedia.org/dashboard/db/mobileapps correctly, then it's even worse than I originally suspected. Only page_summary_-title- is in the black (at just 4:1).

In T201103#4476755, @Joe wrote:

[ ... ]

• What cache hit ratio we have at the restbase layer for MCS-related entities

.
Assuming I understand you correctly, the answer is essentially 100% here, because we pre-generate everything.

https://github.com/wikimedia/restbase/pull/1039 has been committed to issue a revision delete when a new render is stored. This trades some additional write amplification, in the form of redundantly issued deletes, in exchange for more aggressive culling.

Since confidence is high at this point that the increased TTL has solved the original issue identified here, I will close this issue; I've opened T200937: RESTBase k-r-v storage overcommit to address the overcommit

The storage algorithm only evaluates renders for deletion when new renders are stored, and only those that were replaced by another TTL ago (or longer) are candidates for deletion. Likewise, revisions are only evaluated for deletion when a new revision is stored, and only if the corresponding renders were superseded TTL or more in the past. This means that in a perfect world, there will always be at least two renders stored, but possibly (many) more.