+1 for trying this. Thinking out loud:

Uploaded the above to get the ball rolling on a patch. As a starting point it is essentially borrowing the values used for benthos mw accesslog [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 15, 20, 30, 60]

Yes stalling is fine. The original reason for the switch to cfssl was related to adding a SAN to the thanos-fe certificate. That shouldn't be blocked since we can use still cergen for the time being.

In T326657#9126098, @Clement_Goubert wrote:

Hi,

Just FYI, JS and CSS are currently broken on prometheus-{eqiad,codfw}.wikipedia.org due to 401 and 403 errors, with some CORS sprinkled in

@BCornwall fwiw switching from "sli good" to "sli bad" does have the above in mind, namely by working with the small margin-of-error (by switching to calculation to a bad and total metric) instead of against it (attempting to maintain identical good and total metric values). That'd be actionable in the near term and would avoid the negative sli with the exception of edge cases where haproxy is serving 100% errors. With that said, looping in @colewhite and @fgiunchedi for their thoughts and potential alternate alternatives

@lmata could you please confirm if/when ready to proceed with decom of dispatch infra?

In T313229#9117902, @BCornwall wrote:

Closing as dispatch has been ruled out as an option: See T308467 for follow-up discussion of where we're going.

Agree, although let's create a decom task for that as there are some services on the alert hosts to clean up as well

I'm curious if the recent maglev hashing 'mh' inclusion/migration in T263797 provides any improvement here. On paper using 'mh' scheduler should address session stickiness better than 'sh' did.

In T326657#9095309, @fgiunchedi wrote:

I think we'll need two distinct services there (i.e. to be able to change prometheus (the existing internal endpoint) and prometheus-https (the web interface) independently:

# confctl select service=prometheus.* get
{"prometheus2005.codfw.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=codfw,cluster=prometheus,service=prometheus"}
{"prometheus2006.codfw.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=codfw,cluster=prometheus,service=prometheus"}
{"prometheus1005.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=prometheus,service=prometheus"}
{"prometheus1006.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=prometheus,service=prometheus"}

Thanks for the info. With this in mind I'm going to stall this victorops setup task while the details of the desired alerting/paging/team layout are decided. Once that's sorted please update with the desired team name(s) and members and we'll work on setting that up. Thanks!

Hey @bking, thanks for the task! Could you please point me towards the current team roster in order to get the ball rolling for team/account creations?

Thanks for the patch @BCornwall LGTM

AFAIK this is sorted, please reopen if follow up is needed

In T341606#9067700, @BCornwall wrote:

Based on the IRC conversation, wouldn't A have the same issues that we currently have?

Thanks for creating a tracking task! Quickly adding my notes from looking into this:

Untagging observability to table this wrt the kafka-logging cluster for the time being. Will need to revisit the kafka-logging acl config in more detail as part of planning out the kafka upgrade to 3.x

When attempting to manually reproduce the api call that is erroring:

"code": 401,
"message": "API keys are not supported by this API. Expected OAuth2 access token or other authentication credentials that assert a principal. See https://cloud.google.com/docs/authentication",

Looking into alternate credentials

In T340772#8977299, @herron wrote:

Jun 29 17:00:40 alert1001 docker-dispatch[1484]: ERROR:dispatch.incident.messaging:Error in sending welcome email to redacted@wikimedia.org: [Errno 20] Not a directory: '/usr/local/bin/mjml'

Gmail plugin is now enabled using the existing dispatch google cloud credentials (same as used by drive and docs plugins)

Off hand will need to confirm that vo-escalate, vopsbot and the oncall schedule generator will work (or be adjusted to work) normally after this change.

The Grafana hosts are getting tight on disk space, let's bump disk spec for the bullseye hosts to something like 30GB

In T326657#8919271, @BCornwall wrote:

@herron I see there's an unresolved conversation in that patch. Since @Vgutierrez +1ed it before that conversation, I just want to make sure that it is, indeed, ready for merging.

(fixed in T338127)

Checking in this morning, I see both /srv/mw-log/api.log hourly and /srv/mw-log/*.log daily rotations happened. And with the 6to4 relay in place I'm seeing logs arrive on both sides of the "udp tee". I think we're good here!

In T338127#8902131, @fgiunchedi wrote:

Also note that mwlog2002 does not regularly rotate non-api.log logs, yet doesn't suffer from "big log files" issue, suggesting the mirroring/teeing of udp2log traffic to it isn't working as expected

3.21.0 looks promising. Here are my notes from testing on a throwaway VM. Depending on the version, and configuration order the behavior varies.

Checking back on this after rotations overnight, looks like we're not out of the woods yet. Daily rotations are happening once again, however api.log is now being rotated daily as well. Looks like we're hitting https://github.com/logrotate/logrotate/issues/38

looks like logrotate is treating the glob meant to exclude api.log as a literal filename 🤦‍♂️

In T333614#8898294, @MoritzMuehlenhoff wrote:

Is there a task about the udp2log porting work to Python 3, or will that be unnecessary due to T205856?

mwlog1002 has been upgraded to bullseye as well, resolving!

mwlog2002 is up and running now on bullseye. I made a cursory attempt to use python3, but after fixing errors thrown and getting the daemons up and running under python3, it still wasn't writing logs to the filesystem.

END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mwlog2002.codfw.wmnet with OS bullseye

I live edited reuse-lvm-root-4dev.cfg adding this to the bottom of the file, after another reimage the host boots into the os and is accessible from install_console

Mwlog2002 is throwing an error and dropping into grub rescue after reimage with the reuse partitions recipe, going to try and troubleshoot the recipe

I'm not seeing any traffic arriving to mwlog1002 on port 6379, considering this confirmation of no further redis clients of mwlog

In T335424#8873679, @ops-monitoring-bot wrote:

cookbooks.sre.hosts.decommission executed by herron@cumin1001 for hosts: kafkamon2002.codfw.wmnet

• kafkamon2002.codfw.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster codfw to Netbox

• COMMON_STEPS (FAIL)
  • Failed to run the sre.dns.netbox cookbook, run it manually

ERROR: some step on some host failed, check the bolded items above

Good catch thanks, I had to take a look through my history on cumin1001 because I remember decomming these. Turns out ran the cookbook with the -d (dry run) flag enabled 🤦‍♂️ Will re-run these decoms now.

Looking good, I see the first hourly rotated file on disk