In T313751#8388946, @Gehel wrote:

A few comments on the current dashboard:

• a very quick look at Turnilo: the graph look different enough that I'd like to know why the discrepancies
• as discussed, we should define the service as "working" not only when returning HTTP/200, but also when requests are throttled (429) or banned (403)
• we probably need to dig a bit more into other response codes and the dips we see in the graph to understand what they are and if they are problematic (and thus refine our definition of a "working" service)

Comparing grafana to turnilo, the graphs seem more or less aligned as far as the shapes of the graphs are concerned.

Cloudelastic had a few hosts that didn't have the latest heap size applied; most likely the rolling restart was done before sufficient time had passed for puppet to auto-run on each instance.

Moved to Needs Reporting. Usually we leave tickets open but move them to needs reporting and then gehel closes as resolved after he reviews them, but I'll leave this task resolved for now to avoid re-opening it.

Reimaging now

@Papaul Yup per jbond's comment above we're still seeing the RAID issue. Could we try either rebuilding raid with the current disk, or swapping in a new one and rebuilding? (I suspect the latter is necessary but I'm not totally sure)

We tried https://gerrit.wikimedia.org/r/851711 out on wdqs1009 (making exporter Requires= and After= the blazegraph instance); didn't behave how we expected. For example we thought restarting blazegraph would restart the exporter, but that wasn't the case.

Further context for the rsync approach:

Notes from today's SRE meeting where I asked if anyone had any thoughts:

Decom is done: T321243

In T320482#8309495, @Papaul wrote:

@bking this host is out of warranty. If it is a critical host you will have to let us know and request to purchase a disk. Another option is to check also if we have any disk similar from the decommissioned nodes that we can use.
Thanks.

Final step is to open dcops ticket, and then this can be moved to needs reporting.

With respect to recording nginx request responses:

The current approach we're trying to work towards is recording the nginx response codes for requests. That will give us insight into the number of failures we're seeing.

With respect to the SLO itself, our goal is an SLO that captures the promise we make about service availability: namely, that WDQS is available on a best-effort basis. In practice, this means that if an issue arises out of "business hours", it's acceptable to wait until "business hours" to resolve it. For example, in the most extreme case, if the service were to have an outage on a Friday night, we wouldn't be paging anyone to work the night nor the weekend, but come Monday we'd be focusing our efforts on restoring availability as soon as possible. This specific scenario - a multi-day full outage - would of course be quite rare (on the order of a few times a year at most, but generally much less).

Gehel and I met with bblack today.

Somewhat related: https://gerrit.wikimedia.org/r/c/operations/puppet/+/830240

It works!

Per avg(rate(elasticsearch_indices_search_query_total[5m])) by (index), it seems like enwiki_content is pretty much the main source of query load. Also, looking at 4 hosts (2 of the worst offenders in terms of latency, and 2 average latency), we noticed that the 2 worst offenders had 2 enwiki_content each while the 2 average-latency hosts had only one.

I'm curious about what we've seen that indicates that

Elastic likes to pack a lot of the larger index shards (such as commonswiki) onto a single host

Intro (some context for traffic team)

Search team is working on creating an SLI to measure uptime of WDQS. We want our SLI to map as well to the actual user experience as possible, so to that end we're trying to come up with a way to hit WDQS endpoints externally or semi-externally. Ideally the solution would be ambivalent to the underlying pool/depool state of the underlying hosts (translation: if a host is depooled the request won't ultimately route to it).

In T313431#8218238, @RKemper wrote:

ryankemper@mwmaint1002:~/elastic$ cat psi_codfw_masters.lst
elastic2054.codfw.wmnet:9700
elastic2076.codfw.wmnet:9700
elastic2080.codfw.wmnet:9700
ryankemper@mwmaint1002:~/elastic$ python push_cross_cluster_conf.py https://search.svc.codfw.wmnet:9643/_cluster/settings --ccc chi=chi_codfw_masters.lst psi=psi_codfw_masters.lst omega=omega_codfw_masters.lst
seeds=['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']
to_ret={'chi': {'seeds': ['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']}}
seeds=['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']
to_ret={'psi': {'seeds': ['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']}}
seeds=['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']
to_ret={'omega': {'seeds': ['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']}}

Set new seeds for psi to resolve elasticsearch settings alert.

(We'll need to re-do this step w/ the new new values when we expand up to 5 masters today)

https://github.com/elastic/elasticsearch/issues/62445 && https://github.com/elastic/elasticsearch/pull/62730/files

@pfischer Per the above, you might need to log into Wikitech directly. Want to give that a try when you get a chance and then we'll see if you appear in WikiTech users afterwards?

In T316922#8215118, @bking wrote:

Update: Unfortunately, even though @pfischer has created the Wikitech dev account, nothing has changed since @MatthewVernon 's comment above:

sudo check_user [redacted for spam prevention]
WikiTech Users:
	no user found with [redacted for spam prevention]

Gsuit User:
	Primary Email:	[redacted for spam prevention]
	Aliases:
	title:		No title found.
	manager:	No manager found.
	agreedToTerms:	True

I'm not sure if it takes awhile to sync this information. I will reach out to IT in Slack to see if they have any ideas on the Gsuite side of things.

In T286388#8195348, @EBernhardson wrote:

The upstream change is that elastic was upgraded from 6.8 to 7.10.2 on cloudelastic, with production services migrating this week and the next. I had taken a quick look over global-search and it seemed to be avoiding some of the biggest changes (no more mapping types), but i seem to have missed these other details.

For the first few errors, against .ltrstore and .tasks, it looks like elastic has become more strict in how it handles missing fields. In 6.x elasticsearch always allowed you to query unknown fields and it would resolve as equivalent to a match_none query, since there was nothing to match against. It seems elastic 7.x is now being strict and issuing errors.
For this is suspect we would need to change the index glob patterns to use *_content, *_general, *_file to target the wiki specific indices.

For the invalid regex exception this is unexpected, I'm not aware of any changes we had to make on the cirrus side to get our integration tests passing for regex searches. We run our plain searches against the source field using the query_string query though, as opposed to through a match query as seen here. That might be harder to do here though, as it requires having some code that escapes all the things users might do in query_string context. I'll have to do some experimenting to find what is appropriate here.

In T309810#8179500, @RKemper wrote:

Remembered I still need to create dc-ops decom ticket for the 5 eqiad elastic refresh hosts

This has been deployed.

Some quick pros/cons of two possible approaches to getting the SLI metrics: approach #1 is to run a query or set of queries per-dc at a certain frequency, approach #2 is just to run a query on each host at a certain frequency

Remembered I still need to create dc-ops decom ticket for the 5 eqiad elastic refresh hosts