This is caused by high iowait on the cassandra cluster. See T116097 and T124947.

In T125345#1988198, @GWicke wrote:

This is caused by high iowait on the cassandra cluster. See T116097 and T124947.

Thanks. I'm not entirely convinced though: even if you increase the maximum throughput, there will always be a limit. When concurrency passes that limit, the API should be able to return something other than error 500.

In T125345#1988985, @Nemo_bis wrote:

Thanks. I'm not entirely convinced though: even if you increase the maximum throughput, there will always be a limit.

Cassandra performs much much better with SSDs, so the situation you witnessed should be brought down to a minimum after the disk upgrade. You are right that there will always be a limit, but with SSDs we should have enough headroom so that we don't need to optimise prematurely for extreme cases.

When concurrency passes that limit, the API should be able to return something other than error 500.

This problem should disappear soon, as we plan to implement rate limiting in the main RESTBase cluster, cf. T125123: Add rate limiter functionality to service-runner

Hm, yes, the example URL produces no error 500 for me now (I don't know whether the treeviews code changed). Nice! Either way, I linked this report to some users so that they can report if they see it again.

I still don't understand why return 500. An appropriate status code is 429.

In T125345#2294741, @Nemo_bis wrote:

I still don't understand why return 500. An appropriate status code is 429.

That was done in T135240.

Thanks for the bug report, looks like that 10 req/s limit is too high. For now we might not get to tuning it because we're busy trying to just upgrade it to be faster in the first place. But if the situation persists for too long, we'll throttle further.

This issue will be fixed with the new cassandra cluster we are working on, with our current cluster the behaviour you see is the best we can do. Closing this ticket as a duplicate of the main one: https://phabricator.wikimedia.org/T124314

In T125345#2426259, @Nuria wrote:

This issue will be fixed with the new cassandra cluster we are working on, with our current cluster the behaviour you see is the best we can do. Closing this ticket as a duplicate of the main one: https://phabricator.wikimedia.org/T124314

What's the timescale for this fix? I'm still not convinced that increasing capacity will solve the problem:

In T125345#1988985, @Nemo_bis wrote:

even if you increase the maximum throughput, there will always be a limit. When concurrency passes that limit, the API should be able to return something other than error 500.

Well, we are throttling and returning a 429 (too many requests) when we see access above a certain limit. It's just been hard to predict where that limit is for the old cluster because it's so overworked with spinning disks. We rushed it so we could get people pageview data faster, it was a conscious decision, and I think most people think it was a good one.

The new cluster is online, we're loading it with data right now. We're talking lots of Terrabytes that need to get compacted in Cassandra, so it'll take a while, and we can't estimate perfectly, but most likely pretty soon.

Once that's online, we'll load test it and figure out more precisely where its limits are, and throw 429 errors more predictably.

What's the timescale for this fix? I'm still not convinced that increasing capacity will solve the problem:

The ETA to have new cluster in service is end of September. The throughput and throttling thresholds in the new cluster will be higher thus making this error less frequent

Adding a bit more info about the new cluster for @Nemo_bis. The new hosts' SSDs allowed us to load Cassandra with different settings, like compaction set to Leveled rather than DTCS, that will bring more performances and stability. At the moment the major cause of the 500s are timeouts due to long disk seeks, that are slowing down Cassandra.

We tried to apply various workarounds to the current AQS cluster to improve its performances but we kept hitting the same hosts limits. If you want more data I added some thoughts in https://phabricator.wikimedia.org/T143873

Feel free to follow up with us on IRC if you have more questions :)