The bandaid is in place (restart rsyslog.service every 4 hours, the 4 is a magic number, it can be tweaked). Let's see how we go with this in place while a better solution is found

This is done! we're running jaeger collector and query 1.56

In T355963#9700818, @Andrew wrote:

That's correct yeah, I don't think there are security implications. What I'm after is the possibility to upload/config a keypair once and reuse that across instances launch, which at the time I couldn't find a way to do. Perhaps the underlying APIs do support it: once a named keypair has been used once then it can be reused for subsequent launches? That'd be enough for me

Huh, I think I'm having the opposite problem: once I create a keypair and launch a VM with it that keypair is forever associated with my account and installed by default in all future VMs. That sounds like what you want, is that not what you're seeing?

This is done, I've gone for a simpler approach for now to just delete old jaeger indices via curator. If need arises we can do other tweaks like replica / shard count tweaks and so on

My take to have alert group (i.e. the alert name) at the beginning is the following:

• alerts.w.o is "keyed" to alert groups, not individual alerts
• the optional number of alerts that are firing refers to the alert group as a whole, e.g. I find it confusing to have an alert count next to the individual alert (FIRING [4x] <summary> (<alert group>))
• the alert group name already gives a broad indication of what's wrong

I'm optimistically resolving this since logstash.w.o (nowadays opensearch dashboards) is working as expected

I've fixed the issue with the following on centrallog hosts:

I'm optimistically resolving this since we no longer use mod auth ldap

I've chatted with @JMeybohm about this, and since for example a seemingly-related fix (https://github.com/rsyslog/rsyslog/pull/5012/commits/e8ac82e09f930bf99421cc323c24a9dbf215f9da) is present in the Debian testing repos (plus potentially other fixes) I've backported rsyslog to bullseye as 8.2404.0-1~bpo11+1. It is actually a straight backport (no changes needed) although there's a flaky imfile test, which I haven't verified it is equally flaky on sid.

I'm +1 on moving resolved / firing to the beginning and see what the feedback is.

Following up from a chat yesterday:

In T355963#9699633, @Andrew wrote:

<long pause> Thanks for the feedback!

Indeed, I think grafana_labs.certs.yaml as a whole can be ditched

The configuration hasn't changed, though we did upgrade to Bookworm and together with that came a new version of Alertmanager, thus it might be a regression

Yes we'll be trimming the retention more this week @MatthewVernon

Thank you @Jhancock.wm @herron !

In T361706#9685435, @taavi wrote:

Does this need to be private?

Moving off Q4 board since we have hw in capex spreadsheet and it'll be coming next FY

Resolving this since capacity is under control now and we have more coming next FY as per T357747: Capacity planning/estimation for Thanos

I'm moving this off this Q board since I believe it'll happen further down the line

I haven't observed OOMs related to WAL when doing a Prometheus rolling restart today, I'm optimistically resolving the task, though to be reopened if things change

This is done, available space for Prometheus data has been expanded

Done by @Fabfur, thank you!

Indeed, I don't see the alert at https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DAlertLintProblem

In T361566#9680089, @taavi wrote:

In T361566#9680047, @fgiunchedi wrote:

Just a note that we now have https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet#wmcloud.org_zone_delegations - depending on your use case could that replace the floating IP?

Thank you, to clarify what I need is *.o11y.wmcloud.org (HTTPS only) to be answered by an instance/backend. If the generic proxy can do also zone delegation then I'm all for it! Last I checked this wasn't possible, hence the floating IP, though things might have changed

That's indeed supported too, I clarified the docs.

Thank you folks for the quick action on this! Appreciate it

This is done, we're using . as separator

This is done, we have more space for prometheus

I pushed forward with this to be in a stable/known state ASAP, i.e. alert1001 and alert2001 are both on puppet 7 now and catalogs compile successfully

Set batphone for today until Monday COB

Thank you @RobH, I've coordinated with @herron and he'll be helping with this

I've been working on debugging this too, here's my understanding:

• naggen2 is used to generate icinga configuration for nagios_host and nagios_service exported resources, runs as a generator on puppet master/server
• naggen2 reads /etc/puppet/puppetdb.conf to discover the puppetdb url
• on puppetmaster naggen2 works because puppetdb url points on port 8443, which has ssl cert validation as optional
• this is not the case on puppetserver, thus naggen2 can't query puppetdb

Also cc @VRiley-WMF if you could help with this? thank you!

This is fixed, I've undone my symlink bandaid. I've also reported the issue at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067768

Good news and bad news, in the sense that I can't reproduce the OOM in prometheus k8s in codfw, I suspect my fix at https://gerrit.wikimedia.org/r/1013515 to fetch less Envoy metrics significantly reduced load and thus replaying the WAL doesn't pose a memory problem anymore.

Thank you @andrea.denisse for the suggestions! Let's indeed discuss further what are the best options going forward

Promising results, samples/s in eqiad went from ~200k/s to ~110k/s after the change (and slightly increasing)

@Jclark-ctr it looks like one of the new SSDs from {T359452} isn't happy, I've located the drive so it should be blinking; could we replace it ASAP? please ping me on IRC when you can, thank you !

Thank you for the heads up; for context I'm working on T352640: Fix Pontoon to bootstrap from Bookworm and Puppetserver which will enable us to rebuild the whole o11y stack with Bullseye/Bookworm VMs

Thank you @andrea.denisse for taking a look!

This is done, thank you @Papaul

In T359640#9642256, @fgiunchedi wrote:

We (o11y) have brainstormed this issue a little at the offsite, and one partial solution would be to get a prometheus dedicated mw instance, to at least contain the blast radius.

We'll have to brainstorm a little more, though even with moderately-sized histograms I can see statsd-exporter per-pod not being manageable when we're talking big histograms and hundreds of pods

@Jclark-ctr @VRiley-WMF please ping me on irc when you get on site tomorrow and we can coordinate, I'll be around, thank you!

Thank you @Jhancock.wm ! how's tomorrow at 16 UTC for you? we'll be doing both hosts one at a time, and just to confirm: the drives are hot swap (?)

In T359640#9625172, @Krinkle wrote:

Can we disable host-level instance for MediaWiki's statsd exporter? (Or substitute with a constant?) I believe that would save 100x or 2 orders of magnitude. I can't imagine that ever being relevant for service/domain-specific stats from the MediaWiki application. I imagine of the hypothetical use cases that we don't yet have today, 99% would be covered by site="codfw", if we keep that.

Good point re: statsd_exporter_events_conflict_total, looking at a mw-on-k8s world, I think linking the statsd-exporter lifecycle to mw seems the easiest? which also begs the question: maybe it does happen already during mw deployments as pods are cycled?

For awareness, see also https://phabricator.wikimedia.org/T359178#9640223 re: statsv in the context of varnishkafka deprecation/removal.

Yeah having some ballpark numbers will be a great help @cmooney, unless we're talking hundreds of thousands more metrics than we have now I think we're good to go, tens of thousands we can do without much effort/resources

Ah yes indeed, thank you @JMeybohm !

Indeed the WAL grew quite fast (faster than I expected anyways) as the mw-on-k8s migration progressed (we're at ~50% now)

Calling this done, albeit with an hack

Logs from ircecho.service

Thank you @LSobanski ! Those are known, I've silenced the alerts for now, leaving the task open as a reminder