optimistically resolving now that the checklist in the desc is complete

In T349159#9360246, @lmata wrote:

can you help me identify what is relevant to:

• Transfer Grafana alerts (AlertManager).

...I'm hoping we may be able to close this task with your latest patch.

In T352128#9362457, @Volans wrote:

As we got an email from VO about unassigned overrides I think that the issue here is that only one rotation was assigned and not the one that actually pages:

One option that comes to mind is relabeling with something like labelkeep to ingest only the labels we want/need on the prometheus side. That'd let us cut down without modifying the framework at the source.

Prometheus1005 is an R440 which should have 10 total 2.5" bays, today there are (6) 2T SSDs installed. I think it'd be worth getting the ball rolling on adding another (4) SSDs.

I spent some time today experimenting with https://github.com/grafana/cortex-tools, specifically cortextool analyse grafana which looked promising, but unfortunately throws parse errors when it encounters a period in the metric name which makes it not suitable for graphite metrics.

I've had success mounting non-root filesystems that were unreliable (networked fs, external arrays, these kinds of things) using autofs, which these days can be done in systemd.

Thanks for the input everyone! Sounds like we have a consensus on option 1. I'll get started with rolling collector VM reboots into 12GB memory then upload a patch for the JVMs and go from there.

In T350434#9308048, @fgiunchedi wrote:

Would we have enough Ganeti resources to bump all the VMs to 12GB?

Personally I'm for doing option 1 right away, seeing how we handle the next log spike(s) and go from there. One of the main upsides from my view is this option changes the fewest things, essentially only the logstash JVM size as opposed to shuffling services around or introducing new host variants.

After a rolling restart of apache2 cpu utilization is down to essentially 0. Resolving for now, will investigate further if cpu util rises again.

In T350402#9302137, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2023-11-02T14:56:17Z] <herron> logstash1025 systemctl restart apache2.service T350402

MW statsd-exporter instances are being scraped by production prometheus

With the above patch statsd_exporter has been deployed to mw hosts via puppet

In T344751#9293584, @lmata wrote:

In T344751#9292738, @gerritbot wrote:

Change 954114 merged by Herron:

[operations/puppet@production] profile::mediawiki::common: set default histogram buckets

https://gerrit.wikimedia.org/r/954114

Does this mean we're decided? 😃

promtool tsdb create-blocks-from rules is looking promising so far. Here's an slo recording rule that was deployed 2 days ago with history going back 12 weeks:

The initial approach I'm thinking of is running promtool tsdb create-blocks-from rules against the recording rules created by pyrra (since they are new and have no production dependencies yet) and output that into an unused scratch directory. Then transfer that over to pontoon and experiment with loading/using them, and go from there. Sound alright?

Most promising option at the moment looks like https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-tsdb-create-blocks-from-rules

Thank you, this has been useful already!

Perfect, I've updated https://gerrit.wikimedia.org/r/c/operations/puppet/+/954114 to reflect this and I think with a +1 from @Krinkle we'll be good to go.

Thanks SGTM overall, I'll propose just one amendment

Currently maxmemory is set to 1Mb

In T348756#9246345, @hashar wrote:

We have a Redis dashboard but arclamp1001.eqiad.wmnet is not collected there.

Above is a patch for initial audit logging of POST data via modsec. Once we have some example data to work with we can refine the rules to log more human readable entries.

In T341606#9248134, @BCornwall wrote:

@herron Thanks for all of your help. We've implemented varnish_sli_bad. I followed the formulae presented at the top of grafana-grizzly's slo_definitions.libsonnet and got these results. They seem to differ from the current dashboard's values, however. Would you be kind enough to double-check to make sure I'm not missing anything? Thank you. :)

initial inclination/impressions:

Done!

When I raised the idea of using curator for jaeger earlier this year the thinking then was essentially that not managing these indices in curator was a feature.

That's been sorted: RECOVERY - Check systemd state on alert1001 is OK: OK - running: The system is fully operational

We (o11y) have moved pyrra to the titan hosts, which makes this task moot. Transitioning to invalid

GCP: Project "Dispatch" is now shut down and scheduled to be deleted after Oct 25, 2023.

In T346144#9185491, @RLazarus wrote:

I think it wouldn't even need to be "make editable," this only changes the default range for the time picker, right? So you can still use the time picker to punch in the dates of other quarters. Not as clean as having a list to pick from, but no worse than it is now.

Had a quick look at the current file limits for thanos-rule on titan2001, I'm seeing 524k as the current limit

+1 for trying this. Thinking out loud:

Uploaded the above to get the ball rolling on a patch. As a starting point it is essentially borrowing the values used for benthos mw accesslog [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 15, 20, 30, 60]

Yes stalling is fine. The original reason for the switch to cfssl was related to adding a SAN to the thanos-fe certificate. That shouldn't be blocked since we can use still cergen for the time being.

In T326657#9126098, @Clement_Goubert wrote:

Hi,

Just FYI, JS and CSS are currently broken on prometheus-{eqiad,codfw}.wikipedia.org due to 401 and 403 errors, with some CORS sprinkled in

@BCornwall fwiw switching from "sli good" to "sli bad" does have the above in mind, namely by working with the small margin-of-error (by switching to calculation to a bad and total metric) instead of against it (attempting to maintain identical good and total metric values). That'd be actionable in the near term and would avoid the negative sli with the exception of edge cases where haproxy is serving 100% errors. With that said, looping in @colewhite and @fgiunchedi for their thoughts and potential alternatives

@lmata could you please confirm if/when ready to proceed with decom of dispatch infra?

In T313229#9117902, @BCornwall wrote:

Closing as dispatch has been ruled out as an option: See T308467 for follow-up discussion of where we're going.