Yesterday I had a fun adventure through UNIX internals diagnosing why the secondary icinga host winds up with lots of stuck nsca processes and fixed that with r499028 above.

Looks like certbot renews the cert but doesn't restart apache correctly?

The memory address is the same in all of these error reports.

proper graphs with some labels extracted from hardware metadata, plus a thermal headroom 'saturation' graph extracted from syslog mtail

@fgiunchedi we should set this device to 0 weight in the rings, yes? Happy to do the change if you'll review

Talked to @ema and he thinks this is related to T215389 and T216006, which should be fixed by https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/494937/ submitted today.

Thank you @The_RedBurn, that was helpful.

Just saw the new timeout work -- query returned a 500 status after ~60 seconds. Boldly going to call this resolved; of course reopen if there's still more to be done.

Lucas, can you verify that this is resolved?

Resolving this; there's followup work to be done but the 'incident' proper is over.

Turns out those queries weren't actually to execute at all. Here's the highest-cardinality metrics on k8s@codfw prometheus:

In T217715#5005574, @akosiaris wrote:

Also making sure url label can't grow with unbounded cardinality.

How do we enforce that though?

It's almost certainly unnecessary with as little write traffic as our grafanas get, but what I've been doing for my own pre-version-upgrade backups is the following:

Twice in two months we've seen all the logstashen in one cluster 'lock up' at around the same time: stop processing incoming events, huge backlog of socket recv-Q bytes, JVM threads drop, CPU usage plummets, threads blocked on futex().

I'm happy to help in the future, although it will also be a learning exercise for me :)

Looks like -n is/was a gawk option?

@Joe made me aware of the existence of partman configs present on install1002 that are not in Puppet.

This would be an incredibly silly way to do it, but it would be very easy to write a check_prometheus invocation for outgoing network traffic from the machine:

in under 10 minutes after installing rasdaemon on thumbor1004 we also saw one there. that machine is such a consistent performer:

we got one:

@jbond kindly backported the buster version of rasdaemon to stretch. I'm going to attempt installing it on a few stretch hosts that are consistently reporting memory issues

I'm not sure if it's helpful here, or if you know this already, but there is a raid fact in facter that is an array of the types of RAID on a given machine.

OK, but IMO let's keep it passive. icinga can continue to run on icinga2001 for now.

icinga2001 looks stable; go for it Rob

BTW @fgiunchedi authored an incident report at https://wikitech.wikimedia.org/wiki/Incident_documentation/20190208-logstash-mediawiki

It appears that how to make Prometheus node_exporter play nice with rasdaemon is an unresolved issue:
https://github.com/prometheus/node_exporter/issues/986

Thanks @fgiunchedi, that's a good thought! However I couldn't find anything in the SEL for a selection of servers that are currently reporting / have recently reported memory issues:

maybe this will be illuminating for someone -- it is stack traces from the gerrit jvm process at the time it was guzzling CPU

I don't feel nearly well-versed in PHP/PSR-3/Monolog nor the MW codebase to suggest implementations, but it seems to me that Mediawiki itself should be doing some ratelimiting/deduplication of its logging output.

Talked some with @BBlack today, who observed that there are in fact a variety of drivers that back this stuff in the kernel, and that it's very possible we're getting all events delivered to us, but that the counters can be back by firmware which makes its own decision as to whether or not to increment said counters.
I did a bunch of digging and failed to find any obvious correlation between this behavior being observed and kernel version. There is somewhat of a correlation between the approximate 'age' of the platform -- it correlates with newer BIOS version dates, newer server model numbers, more feature flags enabled in CPUs -- but there's nothing definitive there.

Ah yes, sorry to be unclear -- the full extent of the data is definitely too much for Prometheus.

@Papaul when you're back in codfw and have a spare moment, can you please unplug the disk on SATA port A from wmf6653 (in row D), and attempt booting it?
The disk on port B should also have a valid boot record on it.

Grub does seem to be installed on both MBRs, with just minimal differences (going to guess pointing at the different physical disk for /):

1. Overwrote first 512 bytes of /dev/sda and /dev/sdb with zeros
   • Boot automatically fell through disk to using PXE
2. Reimaged system again, with the change for the only_debian setting
3. BIOS boot order port A first
   • booted fine
4. Swapped boot order: port B
   • booted fine!

1. Overwrote first 512 bytes of /dev/sda and /dev/sdb with zeros
   • Boot automatically fell through disk to using PXE
2. Reimaged system using raid1-lvm-ext4-srv-dualboot.cfg
   • Got a warning about the LVM volume group name being already in use; clicked through it manually
3. Installer rebooted system, leaving BIOS disk boot order to port B
   • Hung at "Booting from Hard drive C:"
4. Reset BIOS boot order to port A first
   • Booted fine

1. Installed server with standard raid1-lvm-ext4-srv.cfg partman config
   • Booted fine
2. Went into BIOS and swapped boot order of SATA devices (afterwards, port B first)
   • Server seemed to hang at "Booting from Hard drive C:"
3. back into BIOS; swapped boot order back to original (port A first).
   • Booted fine
4. used install-console to grub-install /dev/sdb
5. rebooted using default boot order
   • Booted fine
6. back into BIOS; swap boot order (port B first)
   • Booted fine!

I only did a quick look, but I couldn't find an existing Prometheus exporter for ulogd. I suppose we could write one, or we could just write an mtail module for its log output.

Assumption 1: the partman-auto-raid directive exactly correlates with our use of Linux software RAID in production.
Assumption 2: in order to have a working grub install on each mirror, software RAID1/10 configs must override grub-installer/bootdev to list all the relevant disks.

In T215033#4925462, @Paladox wrote:

So we just need a http check that checks the website (without checking if the ssl cert is valid or not as we have that check already) and also a load check to make sure the load doesn't go too high otherwise it could cause noticeable impact.

I know very little about debian-installer, but here's a guess based on what I found in the puppet repo:

The timeouts on that check_ssl invocation make little sense to me -- a warning after 60 seconds, but critical after 30? Those seem backwards, and also too high by something like a factor of 4.

If the thing we want to monitor is "Gerrit is responding slowly / not at all", IMO that is the thing we should check. High CPU load is just one of the causes that could result in high latency.

Seems like it is happening again

I did a cumin run across the whole fleet to find hosts that have memory errors in their dmesg buffers, but haven't incremented counters beyond 0.

Ah, forgot to update the task, but at the time @jcrespo and @fgiunchedi and I talked, and Jaime's biggest gripe was that iostat-reported "disk IO utilization" is not a very useful metric: it's the fraction of time that at least one oustanding iop was in the disk's queue. On a server that has any load at all, this metric will generally be "100%" all the time; what you actually care about are stats like "queue depth" and "request latencies".

@Dzahn investigation on the mystery of cp4026 ongoing in T214529

One more observation:

Not sure how I missed this, but fixed now.

In T208524#4896817, @Anomie wrote:

! In T208524#4895821, @daniel wrote:
For number (3), I think the question is when and how, rather than if. One, three, five years?

I would ask "if". What benefits does doing that bring that outweigh breaking compatibility with low-traffic low-effort installations?

@faidon @mark Could one of you approve this request?