Details on test methodology, and some initial results are forthcoming, but to elaborate on comments made elsewhere:

wrk produces latency information that includes 50, 75, 90, and 99p quantiles, for example:

Running 10m test @ http://sessionstore1001.eqiad.wmnet:8080
  8 threads and 1028 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.26ms   24.29ms 472.78ms   96.41%
    Req/Sec     6.00k   611.79     7.69k    78.24%
  Latency Distribution
     50%   30.72ms
     75%   38.03ms
     90%   40.84ms
     99%   98.65ms
  28669438 requests in 10.00m, 3.23GB read
Requests/sec:  47774.78
Transfer/sec:      5.51MB

But that doesn't paint a complete picture. Looking at the distribution of request latency according to the Prometheus metrics (which are consistent with the above), you can see that ~50% of the requests are at or below 10ms (~30% are at or below 2.5ms). Most the remaining are in the 50ms range. Clearly there is something occurring in Kask that adds (an oddly consistent) 40-50ms on roughly half the requests. That will need to be figured out, but this is worth thinking about in terms of what/how we measure latencies.

@Eevans should we break out a separate task for identifying the cause of the extra 40-50ms delay?

Also, have you broken out GET/SET/DELETE requests?

In T221292#5144480, @EvanProdromou wrote:

@Eevans should we break out a separate task for identifying the cause of the extra 40-50ms delay?

We can; If it would be helpful to you

In T221292#5144561, @EvanProdromou wrote:

Also, have you broken out GET/SET/DELETE requests?

I have not.

It seems like where your performance focus is right now, so let's call it out.

Also, is this task done when we're within the boundaries of T211721 or is the task just to have a tool to do measurements?

@Eevans maybe the mystery of the longer requests is just that they're writes instead of reads?

In T221292#5144700, @EvanProdromou wrote:

Also, is this task done when we're within the boundaries of T211721 or is the task just to have a tool to do measurements?

I honestly don't know.

T211721 ought to have been done as a prerequisite to building the service (to inform the implementation). That's why it was opened. It should have been based on our expectations for page latency, perhaps with some cost-benefit element, weighing the hit in latency we were willing to accept in exchange for the feature. Instead, it sort of devolved into an exercise of trying to guess where we'd end up, working backward to plug those numbers into the SLA. If that's the still the plan then yes, I guess we finish this issue and plug the resulting numbers into T211721 for our SLA.

In T221292#5145019, @EvanProdromou wrote:

@Eevans maybe the mystery of the longer requests is just that they're writes instead of reads?

That's not it.

In T221292#5148530, @Eevans wrote:

In T221292#5145019, @EvanProdromou wrote:

@Eevans maybe the mystery of the longer requests is just that they're writes instead of reads?

That's not it.

Cool. Looks like you've got some middleware for prometheus instrumentation in there, so I guess a next step would be figuring out if the difference shows up there, too.

In T221292#5148528, @Eevans wrote:

In T221292#5144700, @EvanProdromou wrote:

Also, is this task done when we're within the boundaries of T211721 or is the task just to have a tool to do measurements?

I honestly don't know.

OK. Let's say that it's done when you've separated out GET/POST/DELETE.

T211721 ought to have been done as a prerequisite to building the service (to inform the implementation). That's why it was opened. It should have been based on our expectations for page latency, perhaps with some cost-benefit element, weighing the hit in latency we were willing to accept in exchange for the feature. Instead, it sort of devolved into an exercise of trying to guess where we'd end up, working backward to plug those numbers into the SLA. If that's the still the plan then yes, I guess we finish this issue and plug the resulting numbers into T211721 for our SLA.

I don't think it makes sense to go backwards from what we have to what we want.

There are some numbers here. They're pretty high, but not as high as the ones you started off with. I think we're closing in on what's acceptable.

And I agree that getting those numbers sooner and clearer would have been great. That's my fault. For future services, I'm going to make sure we have performance expectations ahead of time, rather than after the implementation starts.