This deployed as a NOOP on 0.1 (production) and on 0.2 subcommands are no longer throwing recursion did not resolve errors. I think we're good here

In T324335#8623671, @colewhite wrote:

There are a couple issues.

One is in rsyslog where 50-udp-json-logback-compat.conf loads mmjsonparse and omkafka in conflict with 30-output-kafka.conf.

Feb 16 22:08:12 relforge1003 rsyslogd[2102764]: module 'mmjsonparse' already in this config, cannot be added  [v8.2102.0 try https://www.rsyslog.com/e/2221 ]
Feb 16 22:08:12 relforge1003 rsyslogd[2102764]: module 'omkafka' already in this config, cannot be added  [v8.2102.0 try https://www.rsyslog.com/e/2221 ]

The updated version is now managing SRE/Oncall/Schedule, will keep this open for a bit longer to track any bugs

In T328707#8601420, @fgiunchedi wrote:

In T328707#8597842, @fgiunchedi wrote:

Supporting multiple read sources would be indeed nice with the main benefit of requiring no actions on switchover. I have looked at the code and something else I didn't consider the other day is the added complexity as we would be moving from a single threaded python process to async/multiple threads. Overall I think it'd be manageable though, for example read threads per-redis all writing to a queue and the main thread in charge of reading the queue and serialize writes to files.

So I was curious how this would look like and gave it a try here (WIP but definitely reviewable, contains kinda-unrelated changes too): https://gerrit.wikimedia.org/r/q/topic:multiple-redis

The gist is that reading from pubsub.subscribe(config.get('redis_channel', 'arclamp')) is changed to read from a queue.Queue. The (bounded) queue is fed by multiple redis, each doing pubsub.subscribe. Semantics stay the same, in the sense that if the queue is empty for a certain time then the process exists. In this case however a single redis losing its connection would make the thread die, so redis reconnections are also handled as a side effect, for improved resiliency (e.g. arclamp-log.py will start up and attempt (re)connections to all redises). For ease of deployment purposes the configuration is backwards-compatible with what we have now. Let me know what you think!

The updated dynamic SLO dashboard template and config structure is now live. I think we're good here! If any followup is needed please reopen

I notice that this dashboard is configured to match built-in annotations by tag. I am also able to reproduce the disappearing annotation when the tags field is left empty while adding the annotation, but it does work for me if I apply one of the tags listed in this filter (screenshot from Dashboard Settings > Annotations > Annotations & Alerts (built-in)

Excellent! Thanks for doing the user_auth purge!

In T328784#8592711, @fgiunchedi wrote:

With all that said I think if there's consensus we can:

• save a backup copy of grafana.db
• delete all entries from user_auth table

This should bring back to an expected state, thoughts?

Looking for nearer-term options I found that removing a user from the user_auth table will cause their user entry to no longer show 'synced via oauth' and isExternal attributes, and doing this for user 742 (from the task description) was enough to allow the sync process to complete successfully.

Thanks @jbond that looks ideal, and if we can land on a working config would possibly allow us to simplify the ro/rw domain layout as well.

Updated this dashboard to use phab.* in queries and instance labels in legends so this should continue working across host changes in the future (and support multiple hosts)

Dug into this a bit, and AFAICT the "synced via oauth" and "User info cannot be updated for external Users" both relate back to isExternal:true on the user object. And isExternal:true does appear to be the case for our users populated by the grafana-ldap-users-sync script.

Looping in @KFrancis for NDA confirmation as well

@Ottomata @odimitrijevic could you please review/approve this request for groupadd to analytics-privatedata-users?

Hi @Abhas, the requested access has been provisioned and will fully propagate across the fleet within 30 minutes.

An updated version of the script that retains shift history is now updating https://wikitech.wikimedia.org/wiki/Sandbox-votesting

Potential option 3: Jaeger outputs to kafka-logging as a buffer, jaeger-ingester (perhaps deployed within the logging cluster) reads from kafka-logging and persists to opensearch

In T247517#8211187, @jbond wrote:

• did the emails informing @herron that the machine was due to be deleted go out correctly
• where they received by @herron (spam filter etc)
• why where they not acted upon (possibly not enough notice)

I've drafted an updated template available at the link in the description, and this is being used for creation of new incidents in the production dispatch instance.