This started happening because I added a grouping workaround in the parent task where essentially the grouping is done by puppet instead of pyrra itself. It results in pyrra generating more output config files, e.g. now slo-$site.yaml vs what used to be slo.yaml

Reviewed fs utilization on codfw/eqiad prom hosts and grew their k8s and ops filesystems targeting ~85% free space each

Still seeing two spaces after the status e.g. FIRING: although not seeing a clear cause for that

Overall this dashboard is meant to show graphite utilization for the whole installation, so I think the thing to do is add filters to drill down as needed.

Added two panels at the bottom of the dashboard to display count over time details using the time picker

FWIW we recently added disk capacity to these hosts with about ~1T free in the VG. I've made a note also to discuss/plan next week how best to allocate the additional space for the long-term with the team. In the mean time should it fire again it is safe to grow the LV again, though hopefully it wont be necessary.

fwiw this look quite similar to the diff from the experimental patch where the global cert name was changed to a discovery.wmnet domain:

Thanks! Looks good!

Prometheus1005 is down and depooled, any time works!

Reopening -- today we experienced a memory issue on prometheus1005 which presumably relates to this maintenance. Could we arrange to swap the faulty DIMM outlined in T362990? Thanks in advance!

FWIW I think the current alert text makes sense based on the premise that all alert recipients will/should know about how alerting system internals are structured.

While considering this I'd also like to propose moving the (alert name) to the end of message at the same time. For example:

Hey @VRiley-WMF, I'll help out with this one for the o11y side.

With T352756 T359879 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1017873 in mind I think It'd be worth spending some time here to work out a strategy for bringing backfilled metrics into production.

I think we're in good shape here, please reopen if anything else is needed

In T359879#9669840, @elukey wrote:

@herron something really strange: https://w.wiki/9bMW

I compared the recording rule with the actual metric, trying to aggregate with the same labels, and the results of the recording rule are strange. I see both eqiad and codfw traffic reported, while the original metric shows mostly eqiad traffic. Anything that comes up to mind on Thanos that could cause this?

SSD and RAM upgrades have been installed thanks @Jhancock.wm!

CC from IRC chat -- We've tentatively scheduled this for this Weds afternoon (Eastern TZ, 4/3/2024)

In T361229#9669564, @Jhancock.wm wrote:

I have located and set aside the parts to be installed.

I am available every week day between 1300 UTC and 1700 UTC. Please let me know what time/day in that works best.

Ah excellent! I thought we would have to order new. Yes in that case lets go ahead with 32Gig DDR4 2666 please. Thank you!

scrapping this as ram procurement request for titan has already been submitted

FWIW I just went through a similar triage and broker restart process in T358870

IMO it'd be worth considering a split of the collector role into independent dashboard, logstash and potentially logs-api roles to provide additional isolation between services.

The relocation/rebalance plan outlined in T326419#9639228 has finished running, and a re-run of topicmappr with fresh metrics now shows no proposed moves. Resolving!

Using topicmappr I've generated the below plan to rebalance kafka-logging eqiad:

Renewed certs have been deployed and alerts have cleared, and the auto renew period for Kafka certificates has been increased from 11 to 30 days before expiration.

Spent some time looking into this, and I'm also wondering why these queries seem to yield different results when ran over more than ~15d

In T358870#9591068, @herron wrote:

At any rate, I'll plan to check back on this on Monday and see if the certs have indeed been updated on disk by puppet.

A few observations after looking into this:

@fgiunchedi thanks for the task! sure, having a look into this

Footer is now in place on (slo|pyrra).wikimedia.org

Here's a capture of what was logged by apache in the minute leading up to titan1002 hanging https://phabricator.wikimedia.org/P56453

In T354904#9466544, @lmata wrote:

Based on my understanding of the task, it appears that the only viable way to test it is by implementing it in a live environment. While this may seem spooky, I don't think we have another environment to simulate this effectively to test the sampling.

A custom grafana graphite datasource exporter, and a grafana dashboard using these metrics to outline current graphite datasource utilization have been deployed.

In T354904#9457357, @colewhite wrote:

Allowing Benthos to join Logstash's consumer group is to test the Benthos config and observe the result without interrupting current ingest.

Thanks for the context, although I am still wondering why benthos and logstash would share a consumer group? As I understand it benthos would be ingesting these topics instead of logstash, as opposed to benthos and logstash consuming the topics together.

@colewhite could you expand on the rationale for benthos joining the logstash consumer groups as opposed to using their own 'benthos' consumer groups?

The requested access has been granted and will be live within the next 30 minutes. Transitioning to resolved, but please don't hesitate to reopen if any followup is needed. Thanks!

Great thanks, I've just uploaded the patch. Next step will be a few approvals:

Closing as invalid due to inactivity, please reopen with the requested updates when ready to proceed. Thanks!

In T353958#9423892, @AnnWF wrote:

In T353958#9423872, @herron wrote:

Hi @AnnWF could you please confirm that the developer account name is correct? The account named wfan is not associated with the email address in the description. Thanks in advance!

Ha, just verified, it should be Wfan219, thanks!

Hi @AnnWF could you please confirm that the developer account name is correct? The account named wfan is not associated with the email address in the description. Thanks in advance!