Sounds great, please reach out and/or send reviews if sth is amiss with the standard recipes

Thank you @cmooney for the extended explanation, very much appreciated! Adding @wiki_willy for visibility as dcops is the recipient of InterfaceErrors.

I've bumped the for threshold, resolving for now and will reopen if the issue is back

No reoccurence, resolving

Please excuse the drive-by comment, I've worked with @Muehlenhoff on standardizing our partman recipes and I'm wondering if the standard raid0 recipes (i.e. raid1 for / and raid0 for /srv) would work in this case?

In T347148#9190830, @cmooney wrote:

In T347148#9190724, @fgiunchedi wrote:

ns-recursor VIP uses cloud VPS addressing, correct?

No, it's a service hosted by the cloudservices bare-metal nodes. These were previously using public IP addressing in the WMF production realm for all services, and thus reachable from everywhere.

Now the cloudservices nodes are connected to 10.x addressing (same as any other wmf private host basically), as well as having a leg in a new cloud-only network using 172.20.x.x. Their 10.x IPs can be polled by prometheus no problem. They also announce some public IPs in BGP, for services that need to be available from the internet, which are currently reachable from private WMF space (unless we put ACLs in to block it). They use other, 172.20.x private IPs to host services that are internal to cloud and don't need to be exposed to the outside.

Change deployed, we'll be standing by and see if thanos still laments evaluation failures. Note that prometheus itself has experienced some, although that is completely different and tracked in T347167: Temporary prometheus alert evaluation failures on host role change

Thank you for raising this @taavi !

It isn't thanos-rule itself reporting the error message, but thanos-store that rule talks to; in other words the error message is nested

I'll optimistically call this specific issue resolved, the nail in the coffin will be file-based xds for envoy

I've looked into the puppet logs from the first puppet run on cumin1001:/var/log/spicerack/sre/hosts/reimage/202309130825_filippo_2981305_titan1001.out and the initial failure is because systemd::syslog can't find the envoy user when setting directory ownership:

Test was successful in the sense that we can keep processing the webrequest firehose from codfw (centrallog2002). Even though by taking a performance hit in terms of messages processed, of course also network bandwidth takes a significant hit, jumping ~ +25MB/s. I believe that in the unlikely event of one centrallog host being unavailable for multiple hours we can still get a reasonable webrequest sampled stream. If even that doesn't work for some reason, it is easy to spin up benthos on different hosts since it is all stateless and trivially horizontally scalable

For reference, the dashboards me and @elukey are looking at:

Opened T346759: Investigate and deploy 'max-repeaters = 20' to all librenms devices for followups, this is done

Thank you for reaching out @Urbanecm_WMF and letting us know about metric cleanup! This is done

+ netops for visibility since this can impact network devices

Setting max-repeaters to 20 definitely had an impact on bgp peers poll time:

I tried the setting above on https://librenms.wikimedia.org/device/device=159/tab=edit/section=snmp/ though the web UI reloaded and the text field was empty, suggesting to me that the setting "didn't take"

This is done, webrequest_live is more robust against partial / unindexable requests

That's correct yes, all done

This is done, we're using prometheus-assemble-config for snmp-exporter too now

Sweet, thank you @ayounsi !

This is done

The integration works again, what I did is:

Nothing to do, host was reimaged:

@elukey and I looked into this, turns out that these records have dt set to - and therefore not indexed in druid, current plan is to drop those at the benthos level before sampling

Hosts reimaged with raid0, resolving

@Hokwelum you can find alertmanager onboard documentation for new teams at https://wikitech.wikimedia.org/wiki/Alertmanager . Please add me or other members of observability to the gerrit code reviews and we'll review/merge them for you! Please reach out too if you need further assistance and/or have questions

There's also a related problem, which is more a puppet one, that if the build-envoy-config exec fails (like it happens on the first puppet run) it is never retried unless one of admin-config.yaml or runtime.yaml changes (and the exec is called again)

This is preventing zero-touch reimage of hosts running envoy AFAICS.

First puppet run does indeed create /etc/envoy/envoy.yaml if isn't present, trying to fix its permissions

I was a little too hasty here, I forgot we need raid0 on these hosts to be able to store blocks to be compacted, will need to reimage the hosts

New hosts are in service, resolving

I noticed this because it seems that during said incident with many invalid ip benthos has sent out more messages than I'd have expected:

re: the last point, namely cleaning up thanos components off thanos-fe (therefore leaving only swift) I initially thought of going the state => absent route, though that seems more trouble than removing the thanos profiles from thanos::frontend role and roll-reimage the thanos-fe hosts. What do you think @MatthewVernon ?

In T344136#9159125, @ayounsi wrote:

@andrea.denisse. is there a task for this blocking issue? As more and more people are going to upgrade to bookworm thanks for finding those bugs.

Looks like the alert is working as expected: https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DPuppetConstantChange

mesh tracing for citoid also enabled in staging now!