The dse-k8s prometheus instance is known as k8s-dse on the prometheus hosts themselves. prometheus[1-2]00[7-8] host dse-k8s prometheus instances, so any changes we make should start there.

Thanks Lukasz and Moritz. I'll create the VM myself and release the host into DC Ops inventory as you suggested.

I've merged the above CR, which means we've now satisfied the AC for this ticket. Closing...

I've rebooted all checked hosts and it looks like T431826 was already updated with the reboot progress, so I'll go ahead and close this one out.

Raising priority so I can finish this week.

I just wanted to mention that Terraform/OpenTofu supports other backends besides file, including Kubernetes, S3, http, and postgres. I don't know if this changes the design, but I did want to mention there are options that might be a little more friendly to gitops. Secrets in plaintext are still a concern, but we should audit those and see it's possible to avoid them, perhaps by using OpenTofu's ephemeral support.

@hnowlan Thanks for your support! As far as what DPE SRE needs, would someone on your team be available to do code/documentation reviews? If there is a specific person or people willing to be the Observability contact, let us know. I think we could definitely benefit from y'all's perspective.

Assiging to @Jclark-ctr per conversation in #wikimedia-dcops IRC.

I don't think we've seen this issue in quite some time. Closing...

Matomo is complete; unassigning and pinging ML SREs to look into the liftwing studio rebuild.

The reimage of dse-k8s-worker1042.eqiad.wmnet is in progress, the other 2 hosts can't rename due to redfish errors:

@SCampos-WMF A s Ben mentioned, we won't be able to do this until we migrate to Kubernetes. As such, I've added this request to the AC for T436186 and I'll close this ticket out. If that's not acceptable, feel free to reopen or ping me in IRC/Slack.

I've finished the audit and I don't see any more unused persistent volumes. Closing...

During the course of testing, Erik and I noticed some extremely high I/O wait on dse-k8s-worker100[1-4]. Looking closer at the hardware, it seems that the 1.82 TB LVM PV used by TopoLVM is a virtual drive presented by the hosts' hardware RAID cards, and the devices are comprised of rotational disks:

We've merged the script and it seems to be working well so far. Closing...

After careful consideration from our team, we have decided not to move forward with this project because:

Apologies for the delay. I've reverted the performance governor changes as promised above. Since we aren't seeing any benefit from these changes and the power situation in CODFW is tenuous (see above), I'm closing this one out without making any further changes.

As the AC for this ticket has been completed, I'm closing this one out.

Per Moritz's feedback above, it seems we're all in agreement that we don't need a long-term Kerberos footprint in CODFW. As such, I'm closing out this ticket. The VM discussion continues in T438229.

It's been a long time since we made any headway with this one, and it looks like the 3.x Operator is still maturing. As such, I'm unassigning this ticket and moving back to the backlog.

@MoritzMuehlenhoff I don't see any value to tying this service to a specific server, particularly one that seems broken. I guess I would flip the question around: what do I gain from waiting a month plus for Supermicro support to come back with canned answers, and bouncing the ticket back and forth between DPE/DC Ops/IF? This seems like a perfect use case for VMs to me. Please let me know if I am missing something.

Per this Slack message, it looks like @brouberol fixed this already. I just verified myself, so I'll close this one out.

All above the above hosts have moved to the new VLANs. Closing...

@ayounsi Thanks for helping out with the move-vlan cookbook today! . One thing I noticed after the VLAN change is that there were two IPv6 addresses on all the hosts (example from cirrussearch1120):

We haven't heard back from Luca/IF, so I'm gonna go ahead and close this out. Feel free to re-open if we missed anything!

We've enabled the performance governor on row D hosts (starting at about 18:45 UTC, about 25 minutes before I wrote this). We haven't seen any impact on [[ https://grafana.wikimedia.org/goto/stnr5m?orgId=default | Morelike P95 ]]or CPU pressure stalls since enabling the governor, so the rest of the changes are on hold until we can confirm impact. If there's no measurable impact after a few hours, we will roll back.

Hello DC Ops,

Bumping up priority so I can finish this week.

Moving up in priority so I can finish it this week.

Hey all, just a heads-up that we are going to replace this host with a VM, so DPE SRE no longer needs it for Kerberos. If y'all want to reimage or make any other changes, feel free.

This has been addressed by gerrit change ID Ic21bd45c7b200c70656069d4308b41e7a642d728 . Closing...

I've added the tables to the matomo-next database on matomo1003, so we should be able to start testing the helm chart soon.

@herron Thanks for the questions! We were (and are) still in an early phase of hosting quasi-aaS platforms like OpenSearch, Airflow, etc on dse-k8s so it's possible we missed something.

Per conversation with @BTullis , there are existing integrations for the GeoIP databases.

Thanks Moritz!

As promised, I've assigned the primary role to the krb1004 host. I've verified that changes are still syncing by changing my password and verified the change has propagated to other hosts. As we expected, assigning the primary role did not increase krb1004 's resource usage (the more-impactful client-side changes already took place in T435873).

This project is complete. Thanks to everyone who helped!