Page MenuHomePhabricator
Create Task
Maniphest T397295

Create the dse-k8s-ctrl servers in codfw
Closed, ResolvedPublic
Actions

Description

We are building a dse-k8s-codfw cluster, so we will need two servers to act as the API servers.

They will use the same template as our dse-k8s-eqiad cluster, so it will comprise 2 Ganeti VMs.

Follow the guidelines here to commission tham and set them up: https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/New#Control-plane

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add the new dse-k8s hosts to site.pp so that we can create the VMsoperations/puppetproduction+12 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.Jun 18 2025, 9:58 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 18 2025, 9:58 AM
BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.Jun 18 2025, 9:58 AM
BTullis mentioned this in T397301: Bootstrap the dse-k8s-codfw cluster.Jun 18 2025, 10:18 AM
• Stevemunene claimed this task.Jun 24 2025, 9:25 AM
BTullis edited projects, added Data-Platform-SRE (2025.07.05 - 2025.07.25); removed Data-Platform-SRE (2025.06.13 - 2025.07.04).Jul 4 2025, 3:59 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 4 2025, 4:20 PM
MoritzMuehlenhoff subscribed.Jul 8 2025, 10:33 AM

Please use C and D for these.

gerritbot added a comment.Jul 8 2025, 12:33 PM

Change #1167209 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add the new dse-k8s hosts to site.pp so that we can create the VMs

https://gerrit.wikimedia.org/r/1167209

gerritbot added a project: Patch-For-Review.Jul 8 2025, 12:33 PM
gerritbot added a comment.Jul 8 2025, 12:37 PM

Change #1167209 merged by Btullis:

[operations/puppet@production] Add the new dse-k8s hosts to site.pp so that we can create the VMs

https://gerrit.wikimedia.org/r/1167209

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2025, 1:30 PM
ops-monitoring-bot added a comment.Jul 8 2025, 5:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host dse-k8s-ctrl2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 8 2025, 5:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host dse-k8s-ctrl2002.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 8 2025, 5:58 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host dse-k8s-ctrl2001.codfw.wmnet with OS bookworm completed:

  • dse-k8s-ctrl2001 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507081739_btullis_848937_dse-k8s-ctrl2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 8 2025, 6:07 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host dse-k8s-ctrl2002.codfw.wmnet with OS bookworm completed:

  • dse-k8s-ctrl2002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507081748_btullis_848966_dse-k8s-ctrl2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BTullis closed this task as Resolved.Jul 8 2025, 9:10 PM
BTullis claimed this task.
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.
BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 28 2025, 5:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits