I updated the docs:
https://wikitech.wikimedia.org/w/index.php?title=Puppet&type=revision&diff=1833007&oldid=1828050

First used to validate the change made in https://gerrit.wikimedia.org/r/c/operations/puppet/+/522992 but useful generically.

The backlog in Kafka should clear in just a few more minutes. Closing this; separate issues to be opened later for followup work.

I've started an incident document at https://wikitech.wikimedia.org/wiki/Incident_documentation/20190715-logstash and would appreciate more contributors.

I think we likely want to revisit this.

This is very likely related to T226937: Not possible to server-side upload certain images: "An unknown error occurred in storage backend "local-swift-eqiad""

In T92298#5329217, @Legoktm wrote:

Do we support TLS 1.3 yet? I'm apparently connecting over 1.2 still.

In T227100#5328397, @jbond wrote:

being as its a Friday i thought i would have a go at this however i didn't read the original message correctly and before i refactor i want to double check that you want device\\! and not device\!. I would have thought the first form would fail but i don't know how many levels of encoding/escaping are needed?

Just a note that it happened again today ;)

Wow, that was quick, thanks! Riccardo should have time to do the deploy while I'm on vacation 🙃

I think I just found another one that was missed: https://grafana.wikimedia.org/d/000000545/ganeti

NB that the default limit in Varnish actually allows for slightly fewer than 64 headers -- the first line of an HTTP response is parsed into several pseudo-headers internally, so AIUI there's space for ~59 headers.

I'm told the plan is to move these onto Ganeti in PoPs, so that seems just as good.

So, it looks like that this 500 did in fact come from the application layer... but shouldn't we still be getting more response headers from the edge?

I am unable to create a Discourse account because of this loop.

+1 for the relative simplicity of Thanos (from both a design and deployment
perspective)

Am I alone in feeling like this probably deserves an incident report?

it's just one (not-often-used) link down, not a site down; UBN is unnecessary IMO

Telia reports a 'major outage' and is tracking status of our circuit in case 00993514

My guess is that the beginning of this problem correlates with the beginning of the fetch failures in the first graph panel here:
https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1&from=now-7d&to=now

@Reedy manually ran the global renames that were never queued properly.

Some curious stuff in the monitoring data:

Indeed, thanks @ema ! I talked with @fgiunchedi some about this earlier and we tweaked the wording on the Logstash dashboard to remind users that "Varnish" appearing highly in the "top n Backends" panel is not necessarily reflective of a Varnish issue.

There is a "Nagios Compatible" transport, but it is underdocumented and seems to also only write to a local filesystem path (which is presumed to be a Nagios external command FIFO).

@Marostegui just found something we forgot: the use of Prometheus metrics in Grafana's variable definitions (e.g. by a label_values() query)

SGTM @elukey, thanks!

Andrew, can you (or someone else) advise on rolling out this change for Analytics?

We saw one of these events at 14:48 today and pybal reported fetch failures for -- and wanted to depool -- basically the entire appserver fleet https://phabricator.wikimedia.org/P8551

+1. In general I think it would be a great idea to do a lot more with annotations than we presently do:

Thanks! We now believe this is resolved.

For posterity:

In T223319#5182402, @greg wrote:

Just want to throw out the possibility in the future (future) that some of these underlying tools may change and the unique identifier for that service may no longer align with the unique id in the new service. IOW: some of these cools urls might change ;)

In T197126#5177169, @jcrespo wrote:

It would be nice to have a mockup of the API to test soon (with no production effect except maybe some debug information). That will allow to test automation from scripts we have already. I think that would be step #6 ?

Here's my tentative plan for moving forward with this, including a rollout procedure:

+1 to what @Joe said, and to what @jijiki said. Especially speaking as someone who has been at the Foundation only six months now.

Some quick notes from today's meeting:

Trying out a few things here:

In T222620#5163577, @ema wrote:

Interestingly, there was a memory usage spike right before the host crashed.

cc @mark who I know is about to start looking at hardware requests for the coming FY

We now have IRC alerting based on scraping each prometheus for its process_start_time_seconds metric.

My patches are also stuck in the queue, and I'm seeing teammates manually V+2 their Puppet changes.

It does seem much faster now, thanks @elukey ! Impact of loading 30 days on Prometheus is also minimal now -- modest CPU usage and while there was some increase in RAM consumption over baseline while we were both playing with this, it's not concerning. Thank you :)

Also sorry, I don't have a lot of time left over this week; can take a deeper look next week

I think you should just be able to remove the "custom all value" in the dashboard settings and have it work. In this case Grafana will create its own 'all' value that is simply a regex OR'ing together all the known values, which it looks like it computes based on the cluster=kafka_jumbo hidden variable.

I got tied up with goal work and incident response and have only had a little time to spend on this.

I think the 'real' thing we need to notify on here is when Swift decides it wants to stop using a disk (which it did here)

In T222112#5149757, @Ottomata wrote:

I've modified the Kafka dashboard so that only the Summary Row is uncollapsed bym default. I've also changed the default time range to last 3 hours, rather than last 24.

As documented in T222112#5147131 this didn't actually fix the dashboard at fault in this particular incident, but I've heard from another large-scale Prometheus user (and Prometheus dev) that they've had similar problems and recommend 10M as a value.

I'm pretty sure it is these panels that are responsible for the most Prometheus load

I think https://grafana.wikimedia.org/d/000000607/cluster-overview might have been missed here? I see at least some old metrics being used there, e.g. node_memory_Cached in the "Memory per host" section.