Page MenuHomePhabricator
Create Task
Maniphest T430226

Automate workflow for vlan migrations on k8s worker nodes
Closed, ResolvedPublic
Actions

Description

While re-imaging worker nodes as part of the ServiceOps vlan migration work T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets, we realized that the sre.hosts.reimage cookbook does not support drain and cordon logic needed for k8s worker nodes (by design), since we do worker node reimages in batches using the sre.k8s.roll-reimage-nodescookbook in [0]

As such, we could either implement support --move-vlan in sre.k8s.roll-reimage-nodes or decide on some workaround if we choose not to introduce necessarily interactive changes (out of cookbook homer diffs for example) to the current non interactive workflow in sre.k8s.roll-reimage-nodes.

We could also consider an automated workflow for vlan migrations on k8s worker nodes separate to the sre.k8s.roll-reimage-nodes cookbook altogether. A temporary workaround may be to call the pool-depool-node cookbook [1], reimage with sre.hosts.reimage + --move-vlan, then pool again with pool-depool-node.

Conclusion:

Following some discussion with ServiceOps, we've decided to reuse the current renumber cookbook for vlan migrations on worker nodes, with some changes to the deployment host puppet runs. We considered potentially calling those runs only when the --rename flag is called, however some context in sre.k8s.renumber-node: Run puppet on deploy servers [2] highlights that we run puppet on the deployment hosts to avoid scap failures due to the reimages as well as circumvent the need to sleep while waiting for BGP sessions to establish. Suggesting that we should likely keep these runs as default cookbook behaviour when renumbering nodes with it nonetheless.

We've uploaded a patch [3] to increase the timeout and retry attempts on the deployment host puppet runs. Additionally, sampling through some of the deploy puppet runs on the puppet-board it appears that the average runtime for a host can be anything between 130s to 210s. In conjunction with the retry attempts, this should give us enough room to not fail on those runs consistency.

Action Taken:

The above change has been tested on a series of nodes and merged.

[0] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/k8s/roll-reimage-nodes.py
[1] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/k8s/pool-depool-node.py
[2] - https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1070903
[3] - https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1307872

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
sre.k8s.renumber-node: Increase puppet timeout and attemptsJasmineoperations/cookbooksmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

jasmine_ created this task.Jun 25 2026, 9:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 25 2026, 9:29 PM
jasmine_ updated the task description. (Show Details)Jun 25 2026, 9:30 PM
jasmine_ mentioned this in T421721: Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets.Jun 25 2026, 9:33 PM
jasmine_ updated the task description. (Show Details)Jun 25 2026, 9:36 PM
jasmine_ updated the task description. (Show Details)Jun 25 2026, 9:49 PM
jasmine_ updated the task description. (Show Details)
jasmine_ added a parent task: T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets.Jun 25 2026, 9:56 PM
MLechvien-WMF moved this task from Inbox to Needs Info / Blocked on the ServiceOps board.Jun 26 2026, 9:36 AM
MLechvien-WMF subscribed.

Moving to Needs Info to decide how we prioritize and who is picking it up

jasmine_ mentioned this in T430218: NodeBGPSessionStatusNotEstablished.Jun 26 2026, 4:55 PM
Clement_Goubert subscribed.Jun 29 2026, 10:19 AM

There's the sre.k8s.renumber-node cookbook that does depool/move-vlan/reimage/repool

ops-monitoring-bot added a comment.Jun 30 2026, 1:44 AM

Cookbook cookbooks.sre.k8s.renumber-node was started by jasmine@cumin2002 Renumbering for host wikikube-worker1163.eqiad.wmnet

ops-monitoring-bot added a comment.Jun 30 2026, 1:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host wikikube-worker1163.eqiad.wmnet with OS trixie

jasmine_ renamed this task from Add --move-vlan support for sre.k8s.roll-reimage-nodes.py cookbook to Automate workflow for vlan migrations on k8s worker nodes.Jun 30 2026, 2:05 AM
ops-monitoring-bot added a comment.Jun 30 2026, 2:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host wikikube-worker1163.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1163.eqiad.wmnet (PASS)
    • Successfully cordoned node wikikube-worker1163.eqiad.wmnet
  • wikikube-worker1163.eqiad.wmnet (PASS)
    • Host wikikube-worker1163.eqiad.wmnet depooled from wikikube-eqiad
  • wikikube-worker1163 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606300214_jasmine_3242723_wikikube-worker1163.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jun 30 2026, 2:34 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-30T02:34:34Z] <jasmine_> homer lsw1-d3-eqiad* commit 'T430226'

ops-monitoring-bot added a comment.Jun 30 2026, 3:06 AM

Cookbook cookbooks.sre.k8s.renumber-node started by jasmine@cumin2002 Renumbering for host wikikube-worker1163.eqiad.wmnet completed:

  • wikikube-worker1163.eqiad.wmnet (FAIL)
    • Successfully cordoned node wikikube-worker1163.eqiad.wmnet
    • Successfully reimaged node wikikube-worker1163.eqiad.wmnet
    • Failed to run puppet agent on deployment servers
    • Failed to run puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker1163.eqiad.wmnet
  • wikikube-worker1163.eqiad.wmnet (PASS)
    • Host wikikube-worker1163.eqiad.wmnet depooled from wikikube-eqiad
    • Host wikikube-worker1163.eqiad.wmnet pooled in wikikube-eqiad
  • wikikube-worker1163 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606300214_jasmine_3242723_wikikube-worker1163.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jun 30 2026, 3:06 AM

Cookbook cookbooks.sre.k8s.renumber-node started by jasmine@cumin2002 Renumbering for host wikikube-worker1163.eqiad.wmnet completed:

  • wikikube-worker1163.eqiad.wmnet (FAIL)
    • Successfully cordoned node wikikube-worker1163.eqiad.wmnet
    • Successfully reimaged node wikikube-worker1163.eqiad.wmnet
    • Failed to run puppet agent on deployment servers
    • Failed to run puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker1163.eqiad.wmnet
  • wikikube-worker1163.eqiad.wmnet (PASS)
    • Host wikikube-worker1163.eqiad.wmnet depooled from wikikube-eqiad
    • Host wikikube-worker1163.eqiad.wmnet pooled in wikikube-eqiad
  • wikikube-worker1163 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606300214_jasmine_3242723_wikikube-worker1163.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jasmine_ added a comment.EditedJun 30 2026, 4:17 AM
In T430226#12064706, @Clement_Goubert wrote:

There's the sre.k8s.renumber-node cookbook that does depool/move-vlan/reimage/repool

Thanks Clement!

I've tested on wikikube-worker1163 and confirming that it does the expected depool, move-vlan reimage, repool.

Noting however that the run_puppet_agent_deploy() call failed to run automatically as part of the cookbook so I ran puppet manually on both deployment hosts and prompted the cookbook to skip that step (hence the cookbook failure note above as a result). I will investigate what may have caused that and/or if we can do away with this puppet run in the event that we'd like to extract the functionality here further for a more dedicated workflow.

I suspect that extracting the functionality here may be advantageous considering that we have quite a few worker nodes to go in T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets and that we may possibly benefit from decoupling the two use cases for logging clarity purposes at the very least, although open to thoughts here.

jasmine_ mentioned this in T430609: NodeBGPSessionStatusNotEstablished.Jun 30 2026, 4:32 AM
gerritbot added a comment.Jul 6 2026, 11:56 PM

Change #1307872 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/cookbooks@master] sre.k8s.renumber-node: Increase timeout on deploy hosts puppet run to 600, from default 300

https://gerrit.wikimedia.org/r/1307872

gerritbot added a project: Patch-For-Review.Jul 6 2026, 11:56 PM
Scott_French triaged this task as Medium priority.Jul 8 2026, 12:31 AM
Scott_French added a project: ServiceOps-Upgrades-Hardware.
jasmine_ added a comment.EditedJul 8 2026, 12:50 AM

Following some discussion with ServiceOps, we've decided to reuse the current renumber cookbook for vlan migrations on worker nodes, with some changes to the deployment host puppet runs. We considered potentially calling those runs only when the --rename flag is called, however some context in sre.k8s.renumber-node: Run puppet on deploy servers [0] highlights that we run puppet on the deployment hosts to avoid scap failures due to the reimages as well as circumvent the need to sleep while waiting for BGP sessions to establish. Suggesting that we should likely keep these runs as default cookbook behaviour when renumbering nodes with it nonetheless.

I've uploaded a patch [1] to increase the timeout on the deployment host puppet runs and will test on a set of nodes following the deployment server switchover tomorrow. Additionally, sampling through some of the deployment host puppet runs on the puppet-board it appears that the average runtime for a host can be anything between 130s to 210s. Provided we're only ever running on two deployment hosts at a time, 600 might give us enough room to not fail consistently in case wheres that a puppet run is not already inflight. To account for in-flight puppet runs, we might need to increase it slightly more - although will investigate and update the change accordingly.

[0] - https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1070903
[1] - https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1307872

ops-monitoring-bot added a comment.Jul 8 2026, 3:42 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by jasmine@cumin2002 Renumbering for host wikikube-worker1164.eqiad.wmnet

ops-monitoring-bot added a comment.Jul 8 2026, 3:42 PM

Cookbook cookbooks.sre.k8s.renumber-node started by jasmine@cumin2002 Renumbering for host wikikube-worker1164.eqiad.wmnet completed:

  • wikikube-worker1164.eqiad.wmnet (FAIL)
    • Successfully cordoned node wikikube-worker1164.eqiad.wmnet
    • Failed to reimage node wikikube-worker1164.eqiad.wmnet, sre.hosts.reimage returned 94
  • wikikube-worker1164.eqiad.wmnet (PASS)
    • Host wikikube-worker1164.eqiad.wmnet depooled from wikikube-eqiad
ops-monitoring-bot added a comment.Jul 8 2026, 3:50 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by jasmine@cumin2002 Renumbering for host wikikube-worker1164.eqiad.wmnet

ops-monitoring-bot added a comment.Jul 8 2026, 3:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host wikikube-worker1164.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jul 8 2026, 4:43 PM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host wikikube-worker1164.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1164.eqiad.wmnet (PASS)
    • Successfully cordoned node wikikube-worker1164.eqiad.wmnet
  • wikikube-worker1164.eqiad.wmnet (PASS)
    • Host wikikube-worker1164.eqiad.wmnet depooled from wikikube-eqiad
  • wikikube-worker1164 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607081623_jasmine_663124_wikikube-worker1164.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 8 2026, 5:04 PM

Cookbook cookbooks.sre.k8s.renumber-node started by jasmine@cumin2002 Renumbering for host wikikube-worker1164.eqiad.wmnet completed:

  • wikikube-worker1164.eqiad.wmnet (FAIL)
    • Successfully cordoned node wikikube-worker1164.eqiad.wmnet
    • Successfully reimaged node wikikube-worker1164.eqiad.wmnet
    • Failed to run puppet agent on deployment servers
    • Failed to run puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker1164.eqiad.wmnet
  • wikikube-worker1164.eqiad.wmnet (PASS)
    • Host wikikube-worker1164.eqiad.wmnet depooled from wikikube-eqiad
    • Host wikikube-worker1164.eqiad.wmnet pooled in wikikube-eqiad
  • wikikube-worker1164 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607081623_jasmine_663124_wikikube-worker1164.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 8 2026, 5:04 PM

Cookbook cookbooks.sre.k8s.renumber-node started by jasmine@cumin2002 Renumbering for host wikikube-worker1164.eqiad.wmnet completed:

  • wikikube-worker1164.eqiad.wmnet (FAIL)
    • Successfully cordoned node wikikube-worker1164.eqiad.wmnet
    • Successfully reimaged node wikikube-worker1164.eqiad.wmnet
    • Failed to run puppet agent on deployment servers
    • Failed to run puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker1164.eqiad.wmnet
  • wikikube-worker1164.eqiad.wmnet (PASS)
    • Host wikikube-worker1164.eqiad.wmnet depooled from wikikube-eqiad
    • Host wikikube-worker1164.eqiad.wmnet pooled in wikikube-eqiad
  • wikikube-worker1164 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607081623_jasmine_663124_wikikube-worker1164.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 9 2026, 9:53 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by jasmine@cumin2002 Renumbering for host wikikube-worker1165.eqiad.wmnet

ops-monitoring-bot added a comment.Jul 9 2026, 9:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host wikikube-worker1165.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jul 9 2026, 10:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host wikikube-worker1165.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1165.eqiad.wmnet (PASS)
    • Successfully cordoned node wikikube-worker1165.eqiad.wmnet
  • wikikube-worker1165.eqiad.wmnet (PASS)
    • Host wikikube-worker1165.eqiad.wmnet depooled from wikikube-eqiad
  • wikikube-worker1165 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607092225_jasmine_1017065_wikikube-worker1165.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 9 2026, 10:56 PM

Cookbook cookbooks.sre.k8s.renumber-node started by jasmine@cumin2002 Renumbering for host wikikube-worker1165.eqiad.wmnet completed:

  • wikikube-worker1165.eqiad.wmnet (PASS)
    • Successfully cordoned node wikikube-worker1165.eqiad.wmnet
    • Successfully reimaged node wikikube-worker1165.eqiad.wmnet
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker1165.eqiad.wmnet
  • wikikube-worker1165.eqiad.wmnet (PASS)
    • Host wikikube-worker1165.eqiad.wmnet depooled from wikikube-eqiad
    • Host wikikube-worker1165.eqiad.wmnet pooled in wikikube-eqiad
  • wikikube-worker1165 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607092225_jasmine_1017065_wikikube-worker1165.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jasmine_ updated the task description. (Show Details)Jul 10 2026, 4:03 PM
jasmine_ updated the task description. (Show Details)Jul 10 2026, 4:13 PM
gerritbot added a comment.Jul 10 2026, 4:20 PM

Change #1307872 merged by jenkins-bot:

[operations/cookbooks@master] sre.k8s.renumber-node: Increase puppet timeout and attempts

https://gerrit.wikimedia.org/r/1307872

jasmine_ closed this task as Resolved.EditedJul 10 2026, 4:29 PM
jasmine_ claimed this task.

Resolving as this change has been tested and merged.

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2026, 4:31 PM
bking subscribed.Mon, Jul 20, 9:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits