Where does the 3s measure come from? Is that from the switchover script self-reporting that after trying the per-server.txt list in a loop, it stopped after N iterations when the latency went no longer than 3s? Or is that from something else? Do we know how many iterations it did, and why it stopped? (I assume it meant it started plateauing or fluctuating upward again.)

Is it for any specific wiki that stands out, or were the startup urls for all wikis that slow?

The loop itself is new this year afaik. Perhaps a local minimum would make sense (e.g. at least three times or something like that).

On the other hand, all load.php-specific caches should be warm after the first run since they just go straight into php-apcu and memcached. I'm not sure what would be different between the 2nd and 10th request in that regard.

I assume the per-server stuff runs after the per-cluster stuff, right?

In T285802#7185234, @Krinkle wrote:

Where does the 3s measure come from? Is that from the switchover script self-reporting that after trying the per-server.txt list in a loop, it stopped getting after N iterations when the latency went no longer than 3s? Or is that from something else?

I think it was from mediawiki-cache-warmup's output.

Do we know how many iterations it did, and why it stopped? (I assume it meant it started plateauing or fluctuating upward again.)

3 times, see

Unfortunately I don't think the output itself was logged.

Is it for any specific wiki that stands out, or were the startup urls for all wikis that slow?

I don't remember, sorry.

I assume the per-server stuff runs after the per-cluster stuff, right?

Yes.

Sidenote, it's very easy to test the warmup process in eqiad/passive DC with the --live-test mode so we can iterate on it without needing to wait for the switch back.

Outcome of today's meeting:

• We ran the warmup recipe against Eqiad and it did 3 takes, with the last two takes taking around 18 seconds, of which 5s for the per-appserver tasks, where the average average per-server latency was still 1.5s.
• Running the recipe again did not make the latencies better, still around 18s for the task, and 1.5s on the average latency.

When using curl to make some of the requests manually it was consistently around 0.2s and 0.3s seconds, same when run against a live Codfw server. We could not reproduce any slow request in isolation. But, running the warmup script a few more times did consistenly take around 18s with 10 of the 80 requests taking 2 seconds or more. So it would seem the slow requests happen consistently happening under warmup load, but not outside.

The warmup load does not seem significantly high. We send 80 requests concurrently to a server, which is comparable to normal production off-peak load (e.g. mw254.codfw currently has 60-80 req/s with p95 around 800ms).

Some thoughts:

• Maybe the cache is refusing to warm up for some reason. We noticed that the apcu memory use did not increase between the server being cold and during/after the warmup. In fact, apcu use went down several notches with each run (Grafana dashboard). This was very confusing and we double-checked that it is indeed computed correctly (usage not free), and that it was not reset in-between. We wanted to dump detailed APCu metadata but this the php7adm server function for this appears to be broken currently.
• The fact that latency of one-off requests manually with cURL were fast while warmup remained slow suggested that perhaps the servers were already sufficiently warm. However looking at the overall percentile distribution during the warmup shows clearly that this is not the case (there are lots of slow requests, not just a few outliers). But, I would like to actually run the warmup script against a known-warm server to compare against. Kunal and I will depool a server next week in Codfw (now primary) and run it isolated against that without any restart between depool and "warmup" to confirm for sure how it performs on a server that has been up for a while with live traffic. I'm expecting the average latency to be far lower there (closer to 100ms, not 1.5 seconds).

Next steps:

• T288866: Kunal to fix the apcu dump command in the php7adm server.
• T288867: Timo to start rewriting mw-warmup in Python and add a "single host" command (e.g. instead of running against a whole cluster).
• (This task) Kunal to add a minimum loop count (e.g. at least 6 takes before the cookbook evaluates the latency back-off logic, which is basically what we did last time with 2x3 takes. While we're not consistently seeing that this is useful, it seems 6 takes runs fast enough that it shouldn't matter one way or the other and can always be reduced again later if we need to).

Change 707457 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/cookbooks@master] sre.switchdc.mediawiki: Run the warmup cache script at least 6 times

https://gerrit.wikimedia.org/r/707457

Change 707457 merged by jenkins-bot:

[operations/cookbooks@master] sre.switchdc.mediawiki: Run the warmup cache script at least 6 times

https://gerrit.wikimedia.org/r/707457