Something worth considering here, in addition to the naming scheme for SLIs themselves, is recording metrics that represent the SLOs values themselves (e.g. 0.1 percent)

Thanks @ayounsi it's been re-enabled and puppet has been run

FWIW here's a quick review of current gerrit alerting in case it helps when thinking about checks to include in gitlab monitoring.

mentioning (but not yet linking) some pre-existing SLO tasks T258754 T254916 T256629 T263792

Sorry, I should have clarified this initially, afaict a proxy won't work for this case because logstash configures this at the JVM level and would have unwanted effects on the other inputs and outputs. So I was curious what other approaches might be recommended for this type of outward connection?

Hey @ayounsi, what approach would you recommend for outward connectivity from logstash frontend hosts (logstash1023 for instance) to imap.gmail.com:993?

LGTM thanks @Papaul!

@Papaul sure, sounds good. This host is not yet in production so there will be no prep/depool needed before the re-rack.

Hey @Cmjohnson, @elukey, sure this should be no problem. I've set a reminder in my calendar to stop services on this host ahead of the window, and yup as long as the host/network config stays the same ES should do the right thing when services are brought back up. Would like to monitor it as it comes up though, just shoot a ping when ready. Thanks!

That's really exciting! Yes I'd love do see this happen as well, and am on board with the plan that you outlined. Time will be the main constraint for me right now, but yes let's get it started on prep work and then and if necessary can plan out the more time consuming components for the next Q.

Hey @Cmjohnson, when do you estimate this one will be racked and installed?

I hear you, it depends on the use case a bit, but in general a screen shot or similar (along with saving useful views as visualizations and dashboards) will be more durable in the long-term because, for example, logs will age off after 90d.

Yes /goto/ links will need to be re-created. We have updated the links within the operations/puppet repository, and for things like bookmarks simply log in to logstash.wikimedia.org and search for the dashboard then hit share to obtain an updated /goto/ url.

Was hoping for some feedback on the above patch, but since it's been a few days I've gone ahead and merged it. The listinfo page in this task description looks to have improved to me, in that copy/pasting a sampling of text into a translator gives back a meaningful result. How does it look to you @Mormegil?

In T271123#6729318, @gerritbot wrote:

https://gerrit.wikimedia.org/r/654915

In T234854#6652368, @Krinkle wrote:

It might also have something to do with the logstash-* index names, which seems to be the first field that shows an error when editing those filter bubbles, in which case might just be a configuration issue.

An API key for klaxxon (discussed via IRC, that's what this is going to be used for, see linked task as well) has been created and added to the pwstore file 'victorops'.

I think we can go without it, we plan to replace these older hosts in the near future and also have some logstash refresh hardware that was just ordered. Thanks!

The missing cache pop metrics have been backfilled using the above method and the thanos bucket web viewer no longer shows a gap. I think we're good here!

After some testing, I think this may be a viable approach for backfilling:

Thanks @Papaul!

Copies of the missing blocks have been made into /root/gap_blocks on each of the prometheus pop instances

Apache2 on deployment-logstash03 was erroring with [auth_cas:error] [pid 18928:tid 139767719112768] MOD_AUTH_CAS: CASLoginURL or CASValidateURL not defined.

Hi @KFrancis could you please help @DAlangi_WMF with an NDA? Thanks in advance!

Hi @tmletzko could you please also coordinate a comment from @conny-kawohl_WMDE, @WMDE-leszek, @darthmon_wmde, or @Tobi_WMDE_SW approving this request?

@JanJaquemot could you please also coordinate a comment from @conny-kawohl_WMDE, @WMDE-leszek, @darthmon_wmde, or @Tobi_WMDE_SW approving this request?

Hi @IJethroBT-WMF, the requested access has been granted. I'll transition this to closed now, but please reopen if any follow-up is needed. Thanks!

Hi @STran, you have been added to the wmf LDAP group. I'll transition this to closed now, but please reopen if any follow-up is needed. Thanks!

Hi @Swagoel, the requested access has been granted and will be fully active within 30 minutes. I'll transition this to closed now, but please reopen if any follow-up is needed. Thanks!

Hi @Tobias_Schumann_WMDE-ext, the requested access has been granted. I'll transition this to closed now, but please re-open if any follow-up is needed. Thanks!

Hi @KFrancis, could you please confirm or coordinate an NDA for @tmletzko? Thanks in advance!

Hi @KFrancis, could you please confirm or coordinate an NDA for @JanJaquemot? Thanks in advance!

Hi @STran, for our records could you please give a high level description of what the requested access will be used for? Thanks in advance!

The requested shell and LDAP access has been granted, and will be fully active within 30 minutes. I'll transition this to closed now, but please re-open if any follow-up is needed. Thanks!

In T267817#6627468, @Ottomata wrote:

If the comments say that they are probably true. researchers is kinda outdated. Just analyitics-privatedata-users then.

Hi @KEchavarriqueen, the requested group access has been granted. I'll transition this to closed now, but please don't hesitate to re-open if any follow up is needed. Thanks!

Hi @hnowlan gmodena has been added to LDAP group wmf, and the above patch has been merged. Thanks for that!

Hi @KFrancis, could you please confirm that we have an NDA on file for Tobias? Thanks in advance!

I'll transition this to closed for the time being due to inactivity. When ready to proceed please add a comment of manager approval and re-open the task. Thanks in advance!

Hi @DNdubane_WMF, could you please coordinate obtaining a comment from your manager approving this request?

Hi @KFrancis could you please verify that @Swagoel has a valid NDA on file? Thanks in advance!