It's 100 lines of maintenance, testing, and then there's puppetization. IMO this juice is not worth the squeeze, and work on this would be more effective as contributions upstream to node_exporter.

IMO this is beyond our scope - this is non-trivial work for an issue that's unlikely to rear its head again (we only rarely ever had port clashes after years of inappropriate configuration). It's not really in our scope to extend prometheus either and our work-hours can be better directed elsewhere.

More broadly, I wonder if utilizing existing tooling for this kind of behavior would be worthwhile?

ah, makes sense. It was merged in April, which explains why they were still lurking around then. Thanks!

deployment-cache-(text|upload)08 have been shut down for a few days and things seem to be calm. I've deleted the instances and removed the deployment-cache, deployment-cache-text, and deployment-cache-upload prefixes.

@Xaosflux: That list is of our canonical domains (the most important ones). In sum we have hundreds of domains. :)

Also note that I had to upgrade the BIOS on cp6001 to get it to even properly set the bios values, so if the reprovisioning fails, perhaps an upgrade is in order...

Yay, thanks for the help deploying it. Regarding tofu integration, I'm not seeing any existing states, though I'll admit I didn't look super hard. Is this something that can/should be tracked separately?

@SLyngshede-WMF and @Fabfur: Be advised that we ran into those same annoying PXE boot problems we've had in the past on multiple hosts. We decided to make the partman EFI recipe the default for all cp hosts going forward. This means that we'll need to reprovision the server so it's set to UEFI booting instead of BIOS. I've updated the task description with instructions.

Thanks, @Southparkfan ! I went ahead and moved the floating IPs over. The patch is ready for review/merging. I figure things are working now and a little overlap between the steps is okay, but forgive me if it wasn't.

deployment-cp-text09 and deployment-cp-upload09 are provisioned and ready for service, but I'm not sure how deployment-prep's DNS is set up to switch hosts. @Southparkfan are you aware of how to handle this?

deployment-cp-text09 and deployment-cp-upload09 are up and running now, replacement of the incumbent hosts will be tracked in T436468

+1 on moving some of the uploads over

Thanks for reporting and for the patch! Merged, and now appropriately redirecting to es.wp.o

We had a similar issue today: https://en.wikipedia.org/wiki/User_talk:BD2412#Template_move_issues

Thanks for hanging in there, everyone, and thanks @RobH for taking care of the arduous task of getting the RAM over!

Given that removal of the "dead host" flag from the Archive team made it work again it appears that's our cause. The question is then whether we're sending 503s/999s to them. Looking at the graphs, we are not - the number of 503s is imperceptible. As far as I know, we do not emit 999s anywhere. I wonder if the Archive team would be interested in reaching out so that we can get some more information. As things stand we aren't blocking anything and it appears that some automated system they have might need some tweaking. @Cyberpower678 would you either be willing to relay things or get us in touch with this Archive admin you've been speaking to? Thanks!

A suggestion from @ssingh is to include a "quick fix"/"easy win" column a la infra foundations: I'd probably make inclusion of "easy wins" as part of the prepping/gardening process so that they're mixed in with the more "normal" tickets. i.e. The "Queued for service" column would contain the work that we're committed to/wanting to work on in addition to small nuggets to get through at any time.

I would think that using existing alerting infra might be more appropriate (Prometheus/Alertmanager). Consistency is one reason, but also routing capabilities for future expansion/targeted alerts.

We've kept the domain registration in markmonitor but moved the NS to porkbun, which is properly redirecting now.

MarkMonitor offers our edge case as a solution for $3,500 a year. Yikes. We'll be exploring other avenues - My recommendation is for using our AWS account and Route53+CloudFront/S3 for redirection for pennies a year.

Frustratingly, restarting acme-chief's nginx.service fixed it. Don't ask my how or why that worked, but it was documented as a known issue. yikes.

killed this instance in lieu of new naming scheme (https://phabricator.wikimedia.org/T280393)

Not sure if that's related to my work, but I can confirm that deployment-puppetserver-1 can pull that repo.

Not sure if corto is owned by o11y, forgive me if not.

Fixed with https://gerrit.wikimedia.org/r/plugins/gitiles/labs/private/+/cdc68fb6637c27d5893879d5c65478bc68845f26

I just created these instances, as they're going to be replacing 05/06 (bullseye→bookworm)

Bless you, Rob.

Thanks for the clarification. Closing.

The issue we had stems from the existing certificate having been issued from before google pulled OCSP support, so it was checking for a non-existent property. While technically this is an edge case that acme-chief could handle, it's quite niche and IMO not worth pursuing.

What we have is good enough

I also think this can be closed as there aren't any plans (that I know of) to remove the lang subdomains. mobile domains have since been unified. Thoughts, @ssingh?

Hi, @isarantopoulos! Thanks for bringing this to our attention.

A good reason to continue to support TLS 1.2 for longer than e.g. 1.0/1.1 was supported is that 1.3 has stronger protections against downgrade attacks via downgrade sentinels (though it's not perfect). In that respect, we can make a prediction that keeping 1.2 around for longer does not expose clients to such attacks like keeping 1.0/1.1 around did.

I think a few things could be helpful for us:

Please also note that the deprecation takes a long time, too.

@Joe Thanks for the feedback. I've since reworked it - would you mind a second look? Thanks.

Friendly poke: I have https://gerrit.wikimedia.org/r/c/operations/dns/+/1306727 tee'd up.

Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1320759

@EMill-WMF & @Rsilvola: There's been some interest about the current state of things and our timeline. When should we proceed on deprecating/removing TLS 1.2 support?

Here's the results of 7 days of a text node (cp3073):

Confirming I can do those times (2026-07-29 @ 16:00-20:00 PDT)

These are just testing hosts that haven't been in production for a while. lvs1017-1020 are the current production hosts. I've made a note for traffic to discuss whether we can decomm these hosts this upcoming tuesday. For now, these errors can be ignored. Thanks!

@ssingh I am willing to be available on my Monday evening/Tuesday Singapore time.

@pmiazga No preference! That sounds good to me.

cp3073 (text) is now running on a single drive as of this comment. Will check back in a week to compare.

Heh, yeah, forgot this existed. Declining for now, we're keeping dns-related items on gerrit. Thanks!

My understanding of the situation: