Page MenuHomePhabricator
Create Task
Maniphest T343996

Move codfw thumbor hosts to kubernetes cluster
Closed, ResolvedPublic
Actions

Description

Thumbor's bare metal days are over, and it is time to have any (young) thumbor servers join our kubernetes cluster.

DC-Ops ops-codfw

  • Physically re-label thumbor2005 to kubernetes2055
  • Physically re-label thumbor2006 to kubernetes2056

serviceops

  • Remove from puppet
  • update netbox
  • Follow the usual new server process

Details

SubjectRepoBranchLines +/-
kubernetes: add new nodesoperations/puppetproduction+14 -0
kubernetes: add new kubernetes nodes to calicooperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

jijiki created this task.Aug 10 2023, 3:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 10 2023, 3:35 PM
jijiki updated the task description. (Show Details)Aug 10 2023, 3:35 PM
jijiki added a parent task: T334488: Final steps for fully-Kubernetes Thumbor.
Maintenance_bot added a project: SRE.Aug 10 2023, 3:45 PM
Jhancock.wm updated the task description. (Show Details)Aug 11 2023, 2:43 PM
Jhancock.wm subscribed.

servers have been physically relabeled.

JMeybohm mentioned this in T277876: Reserve resources for system daemons on kubernetes nodes.Aug 21 2023, 8:32 AM
hnowlan subscribed.Aug 21 2023, 10:01 AM
Stashbot added a comment.Aug 21 2023, 2:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-21T14:14:21Z] <hnowlan@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: hnowlan: updating records for reuse of thumbor servers for k8s nodes T343996 T343993 - hnowlan@cumin1001"

Stashbot mentioned this in T343993: Move eqiad thumbor hosts to kubernetes cluster.Aug 21 2023, 2:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-21T14:15:10Z] <hnowlan@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: hnowlan: updating records for reuse of thumbor servers for k8s nodes T343996 T343993 - hnowlan@cumin1001"

ops-monitoring-bot added a comment.Aug 22 2023, 11:36 AM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host kubernetes2055.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2023, 11:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host kubernetes2056.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2023, 12:54 PM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host kubernetes2055.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2055 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 22 2023, 12:56 PM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host kubernetes2056.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2056 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
Stashbot added a comment.Aug 22 2023, 2:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T14:14:43Z] <hnowlan@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "refreshing kubernetes205[56] kubernetes105[78] status T343996 T343993 - hnowlan@cumin1001"

Stashbot added a comment.Aug 22 2023, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T14:15:48Z] <hnowlan@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "refreshing kubernetes205[56] kubernetes105[78] status T343996 T343993 - hnowlan@cumin1001"

ops-monitoring-bot added a comment.Aug 22 2023, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host kubernetes2055.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2023, 2:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host kubernetes2056.codfw.wmnet with OS bullseye

gerritbot added a comment.Aug 22 2023, 3:01 PM

Change 951522 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] kubernetes: add new kubernetes nodes to calico

https://gerrit.wikimedia.org/r/951522

gerritbot added a project: Patch-For-Review.Aug 22 2023, 3:01 PM
ops-monitoring-bot added a comment.Aug 22 2023, 3:04 PM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host kubernetes2055.codfw.wmnet with OS bullseye completed:

  • kubernetes2055 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308221446_hnowlan_394988_kubernetes2055.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Aug 22 2023, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host kubernetes2056.codfw.wmnet with OS bullseye completed:

  • kubernetes2056 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308221449_hnowlan_398030_kubernetes2056.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Aug 22 2023, 3:08 PM

Change 951524 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] kubernetes: add new nodes

https://gerrit.wikimedia.org/r/951524

gerritbot added a comment.Aug 22 2023, 3:15 PM

Change 951522 merged by Hnowlan:

[operations/puppet@production] kubernetes: add new kubernetes nodes to calico

https://gerrit.wikimedia.org/r/951522

gerritbot added a comment.Aug 22 2023, 3:18 PM

Change 951524 merged by Hnowlan:

[operations/puppet@production] kubernetes: add new nodes

https://gerrit.wikimedia.org/r/951524

Maintenance_bot removed a project: Patch-For-Review.Aug 22 2023, 3:29 PM
hnowlan closed this task as Resolved.Aug 22 2023, 4:55 PM
hnowlan claimed this task.
hnowlan updated the task description. (Show Details)
Jhancock.wm closed subtask T344597: Decommission thumbor200[34] as Resolved.Aug 25 2023, 1:36 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits