Perfect, thank you! I started https://wikitech.wikimedia.org/wiki/Incidents/2023-02-11_logstash_latency and filled in what we know so far (as far as I know, anyway).

@akosiaris and @Clement_Goubert will come up with a cluster layout this week, and @Clement_Goubert wanted to try putting at least one or two into service themselves. Feel free to assign to me afterward to churn through the rest.

Adding @CDanis as we were just talking about something along these lines.

Sure, we could look at adding a warmup step to the server repool process. Historically we haven't worried about it, because the impact for one host is much smaller than when the entire cluster is cold, but it's worth looking at. I'd rather make this change in-place first though, and then it would be easy to also install something on each host. (We might choose to wait and do this after mw-on-k8s, but we can take a look at it for sure.)

We decided we'll put these into service after the upcoming DC switchover, so we'll make a plan at the March 6 serviceops meeting.

Perfect, thanks!

@Krinkle Are you aware of any current uses of warmup.js besides the DC switchover automation? Anywhere else I need to maintain compatibility, or adapt either humans or software to call the new script?

POSTs will probably have to wait for the Python rewrite, but then they'll be easy. Can you recommend specific requests?

I'm working on this.

In that case it sounds like yes, we do need cache warming in eqiad before repooling it -- and we'll need to add URLs to warm up s8, per this task.

Agree we don't need it in order to switch the RW site to codfw.

This is deployed! Thanks again for the patch, let me know if you need anything else.

(Back from vacation, sorry for the delay.) Yeah, I think we can close this. Thanks!

Thanks @Volans. DC ops, for mw1320, I wasn't able to manually shut it off -- please do just kill the power when you go in to unrack it. Thanks!

Over to dcops!

Oops, yes, that should have been 306162. Thanks for the catch!

In T323707#8419423, @Joe wrote:

Maybe if the page we're trying to fetch is that cumbersome, we should switch to a different, lighter one?

Changed my mind on this -- still going to look into other solutions, but I did bump the deadline to 60s so that it doesn't spuriously alert in the meantime.

Good find, thanks.

Thanks John!

Draft: https://wikitech.wikimedia.org/wiki/Incidents/2022-10-15_s6_master_failure

I was surprised to see SHOW PROCESSLIST didn't empty out after "very few minutes" like wt:MariaDB/troubleshooting#Depooling_a_replica suggests -- for the record, here's what it looked like, in full, 15-20 minutes after @Joe depooled it: P35486

Oh, one more angle to think about! There are two Maps services we're writing SLOs for. For Kartotherian, we're just planning a request latency target at the 50th and the 95th -- i.e, the Prometheus query is the same except for the percentile. But for Tegola, we're measuring two different types of latency (HTTP request latency vs. cache operation time), each of those at p50 and p95. So we'd end up writing different queries, too.

I was wondering if https://gerrit.wikimedia.org/r/c/operations/puppet/+/831230 might be related but I couldn't work out exactly what's going on here.

T312947 already tracks the larger question of how to organize runbooks for ProbeDown effectively.

That sounds right to me; it would give us the same distribution as codfw, which is probably as much work as we need to do on this. I don't think it's worth investing time into any deliberate benchmarking, but if the cluster happens to saturate unevenly again in a future incident, we can tweak opportunistically.

!log rzl@cumin2002 START - Cookbook sre.hosts.remove-downtime for mc2038.codfw.wmnet
!log rzl@cumin2002 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mc2038.codfw.wmnet

Due to T309956 I'm moving ahead with mc2038 early, and using it to replace mc2024 which is currently out of service.

Yep.

Marking this resolved, thank you very much!

@Papaul All yours!

N.B. this is only seven hosts, mw225[1-5,7-8] -- mw2256 was already decommed in T263065.

Hi @CRoslof -- adding you here, per my email just now.

With https://gerrit.wikimedia.org/r/813924 we ought to see smaller bursts in utilization, so I'm going to tentatively crank the shellbox replicas back down to 8, where it was before https://gerrit.wikimedia.org/r/803953.

Yep, looks much better! Closing.

I'd like to redo @Legoktm's manual test first, and make sure I can't reproduce a spike -- I'll do that later today, and close afterward.

In T255511#8064532, @tstarling wrote:

I can't find any incident documentation for an incident on 2020-06-08, and I'm unclear on what problem was caused by mcrouter flapping. Was mc1029 slow, able to serve VERSION probes, but unable to serve a significant amount of real traffic? So while mc1029 was pooled, requests were handled slowly or gave errors?