New hosts added in puppet. Their weights have been set in pybal (more specifically, etcd via conftool), and they're currently marked inactive while we do data xfers. About to kick off batch #1 shortly and then will do batch #2 after the first batch is all finished:
Current status:
Been seeing some weirdness on elastic1107 (internal search team alerts for PuppetZeroResources and the like) so we'll see if a fresh reimage smooths things over
Here's some extra notes with some of the commands we ran/used: P54284
Alright, I had an initial meeting with Traffic team (Brandon & Valentin).
Here's a graph of some of the alerts firing over time. We do see occasional alerts for these new alerts, but it seems less frequent (and therefore more granular) than with the previous alerting strategy (alerting on generic p95 request time rather than breaking it down per query type)
Talked to @EBernhardson last week and one thing we were uncertain of is if it made sense to set SLOs on metrics such as MediaSearch latency p95 which, with the metric representing the actual time to render the view and not just the Elasticsearch backend response time, means that its possible for the SLO to be missed in the absence of there being anything wrong with Search services in particular. After talking with @Gehel, one thing we discussed is that some of these SLOs should be set in terms of what user experience we feel is acceptable; in this way we'll have a metric/objective that we can point to if, say, some change elsewhere in the stack leads to slowdowns. It's sort of analogous to the role unit tests play in refactoring: allowing you to make changes while being able to validate that the changes didn't break something.
Balthazar and I met last week. We took a look at the temporary dashboard and outlined some alerting threshold values for the various SLIs:
Alert changes have been merged so sticking in blocked/waiting for now. We should check back in a week and make sure the alerts work as intended.
Created T348696 for the Illegal Reflective Access warning
Reboot complete. Systemd units are happy.
These hosts are in service as of yesterday.
We've also applied the fix to codfw and made the thresholds equal between both approaches since they're back to using the same metric again.
Okay, @EBernhardson, @bking and I think we've found an approach that actually works:
Just to explain what happened here, we switched the ldf endpoint from 1003 to 1016, which at the time was a public host. However for separate reasons we later realized that 1016 was supposed to be an internal host, and reimaged as such without remembering that it was the new LDF host.
Running decom cookbook for wdqs100[3,4]. Dc-ops ticket up here: https://phabricator.wikimedia.org/T346699
Removed subtask because I think the scap ticket is not directly related to this one.
The reindex, even though we had to terminate it before it finished, had already gotten to enwiki_content. So this is done. Here's what the value on both eqiad and codfw looks like:
Host was reimaged but patch wasn't yet merged. Merging patch and rolling the re-image again.
Oops, meant this to be in progress. Changing status to Open from previous state of Resolved.
We made a slight mistake: one of these hosts needed to be in wdqs-internal since wdqs1003 (one of the hosts replaced by these new hosts) is.
Made the decom ticket for wdqs1005 and ran the cookbook. We will decom 1003/1004 later. Moving this to blocked/waiting for now.