Page MenuHomePhabricator
Create Task
Maniphest T418259

wikikube-worker13[35-59] implementation tracking
Closed, ResolvedPublic
Actions

Description

This task is to track the service implementation of ServiceOps new host(s) listed in the task description.

Once the linked racking task has been resolved, this task can be implemented.

  • wikikube-worker[1335-1349].eqiad.wmnet (was blocked by T411054: Nokia SR-Linux DHCP Relay Bug)
    • trixie (14): wikikube-worker[1335-1346,1348-1349].eqiad.wmnet
    • bookworm trixie (1): wikikube-worker1347.eqiad.wmnet
  • wikikube-worker[1350-1359].eqiad.wmnet good to go
    • trixie (2): wikikube-worker[1350-1351].eqiad.wmnet
    • bookworm trixie (8): wikikube-worker[1352-1359].eqiad.wmnet

TODO: Add add_k8s_nodes.py output as checklist

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wikikube: Add wikikube-worker[1335-1349].eqiad.wmnetoperations/puppetproduction+30 -6
k8s.pool-depool-cookbook: Handle calicoctl exiting with erroroperations/cookbooksmaster+22 -8
Add wikikube-worker[1352-1359]operations/puppetproduction+18 -2
BGPPeers: Add comment for eqiad E4operations/deployment-chartsmaster+1 -0
BGPPeers: Add missing lsw1-f8-eqiadoperations/deployment-chartsmaster+15 -0
Add wikikube-worker[1350-1351]operations/puppetproduction+8 -4
conftool-data: Fix YAML syntaxoperations/puppetproduction+30 -30
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT408752 Q2:rack/setup/install wikikube-worker1335-59
 ResolvedJMeybohmT418259 wikikube-worker13[35-59] implementation tracking

Event Timeline

Clement_Goubert created this task.Feb 24 2026, 3:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 24 2026, 3:36 PM
Clement_Goubert triaged this task as Medium priority.Feb 24 2026, 4:07 PM
Clement_Goubert moved this task from Inbox to Scheduled (this Q) on the ServiceOps new board.
JMeybohm updated the task description. (Show Details)Mar 2 2026, 12:57 PM
JMeybohm claimed this task.Mar 2 2026, 3:52 PM
JMeybohm moved this task from Scheduled (this Q) to In Progress on the ServiceOps new board.
JMeybohm subscribed.

I'll take these to start with trixie in prod

JMeybohm updated the task description. (Show Details)Mar 2 2026, 3:53 PM
gerritbot added a comment.Mar 2 2026, 4:25 PM

Change #1247099 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] conftool-data: Fix YAML syntax

https://gerrit.wikimedia.org/r/1247099

gerritbot added a project: Patch-For-Review.Mar 2 2026, 4:25 PM

Change #1247100 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add wikikube-worker[1350-1351]

https://gerrit.wikimedia.org/r/1247100

gerritbot added a comment.Mar 3 2026, 8:52 AM

Change #1247099 merged by JMeybohm:

[operations/puppet@production] conftool-data: Fix YAML syntax

https://gerrit.wikimedia.org/r/1247099

gerritbot added a comment.Mar 3 2026, 8:52 AM

Change #1247100 merged by JMeybohm:

[operations/puppet@production] Add wikikube-worker[1350-1351]

https://gerrit.wikimedia.org/r/1247100

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2026, 9:30 AM
gerritbot added a comment.Mar 3 2026, 10:36 AM

Change #1247552 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] BGPPeers: Add missing lsw1-f8-eqiad

https://gerrit.wikimedia.org/r/1247552

gerritbot added a project: Patch-For-Review.Mar 3 2026, 10:36 AM
gerritbot added a comment.Mar 3 2026, 11:02 AM

Change #1247552 merged by jenkins-bot:

[operations/deployment-charts@master] BGPPeers: Add missing lsw1-f8-eqiad

https://gerrit.wikimedia.org/r/1247552

gerritbot added a comment.Mar 3 2026, 11:05 AM

Change #1247561 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] BGPPeers: Add comment for eqiad E4

https://gerrit.wikimedia.org/r/1247561

ops-monitoring-bot added a comment.Mar 3 2026, 11:16 AM

Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1003 pool for host wikikube-worker[1350-1351].eqiad.wmnet completed:

  • wikikube-worker[1350-1351].eqiad.wmnet (PASS)
    • Host wikikube-worker[1350-1351].eqiad.wmnet pooled in wikikube-eqiad
JMeybohm updated the task description. (Show Details)Mar 3 2026, 11:18 AM
gerritbot added a comment.Mar 3 2026, 11:24 AM

Change #1247561 merged by jenkins-bot:

[operations/deployment-charts@master] BGPPeers: Add comment for eqiad E4

https://gerrit.wikimedia.org/r/1247561

ops-monitoring-bot added a comment.Mar 3 2026, 11:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 3 2026, 11:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2026, 11:30 AM
ops-monitoring-bot added a comment.Mar 3 2026, 11:31 AM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1354.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 3 2026, 11:31 AM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1355.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 3 2026, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1352 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603031148_jayme_2861527_wikikube-worker1352.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 3 2026, 12:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1354.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1354 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603031152_jayme_2863709_wikikube-worker1354.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 3 2026, 12:14 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1355.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1355 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603031158_jayme_2863943_wikikube-worker1355.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 3 2026, 12:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1356.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 3 2026, 12:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1357.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 3 2026, 12:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1358.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 3 2026, 12:19 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1353 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603031202_jayme_2862711_wikikube-worker1353.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 3 2026, 12:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1359.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 3 2026, 12:48 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1356.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1356 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603031233_jayme_2939014_wikikube-worker1356.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 3 2026, 12:52 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1357.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1357 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603031236_jayme_2939427_wikikube-worker1357.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 3 2026, 12:59 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1358.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1358 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603031243_jayme_2940136_wikikube-worker1358.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 3 2026, 1:10 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1359.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1359 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603031253_jayme_2971945_wikikube-worker1359.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 3 2026, 2:28 PM

Change #1247617 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add wikikube-worker[1352-1359]

https://gerrit.wikimedia.org/r/1247617

gerritbot added a project: Patch-For-Review.Mar 3 2026, 2:28 PM
gerritbot added a comment.Mar 3 2026, 2:32 PM

Change #1247617 merged by JMeybohm:

[operations/puppet@production] Add wikikube-worker[1352-1359]

https://gerrit.wikimedia.org/r/1247617

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2026, 3:31 PM
gerritbot added a comment.Mar 3 2026, 3:35 PM

Change #1247628 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/cookbooks@master] k8s.pool-depool-cookbook: Handle calicoctl exiting with error

https://gerrit.wikimedia.org/r/1247628

gerritbot added a project: Patch-For-Review.Mar 3 2026, 3:35 PM
JMeybohm updated the task description. (Show Details)Mar 3 2026, 3:43 PM
JMeybohm mentioned this in T417772: wikikube-worker23[32-56] implementation tracking.Mar 3 2026, 3:48 PM
gerritbot added a comment.Mar 4 2026, 8:53 AM

Change #1247628 merged by jenkins-bot:

[operations/cookbooks@master] k8s.pool-depool-cookbook: Handle calicoctl exiting with error

https://gerrit.wikimedia.org/r/1247628

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2026, 9:32 AM
JMeybohm changed the task status from Open to Stalled.Mar 5 2026, 11:50 AM
JMeybohm moved this task from In Progress to Scheduled (this Q) on the ServiceOps new board.Mar 5 2026, 3:55 PM
MLechvien-WMF moved this task from Scheduled (this Q) to Needs Info / Blocked on the ServiceOps new board.Mar 12 2026, 9:05 AM
JMeybohm changed the task status from Stalled to Open.Mar 17 2026, 3:51 PM
JMeybohm updated the task description. (Show Details)
Scott_French moved this task from Needs Info / Blocked to Scheduled (this Q) on the ServiceOps new board.Mar 18 2026, 1:55 AM
Scott_French subscribed.

@JMeybohm - I'm speculatively moving this to Scheduled based on your most recent update. If it's something that you plan to pick back up now that it's unblocked, please move back to In Progress. Thanks!

ops-monitoring-bot added a comment.Mar 18 2026, 4:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host wikikube-worker1347.eqiad.wmnet with OS trixie

JMeybohm moved this task from Scheduled (this Q) to In Progress on the ServiceOps new board.Mar 18 2026, 4:45 PM
In T418259#11721443, @Scott_French wrote:

@JMeybohm - I'm speculatively moving this to Scheduled based on your most recent update. If it's something that you plan to pick back up now that it's unblocked, please move back to In Progress. Thanks!

Yeah, sorry. Missed moving this one.

JMeybohm updated the task description. (Show Details)Mar 18 2026, 5:03 PM
ops-monitoring-bot added a comment.Mar 18 2026, 5:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host wikikube-worker1347.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1347 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603181719_jayme_74183_wikikube-worker1347.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 19 2026, 11:55 AM

Change #1255689 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] wikikube: Add wikikube-worker[1335-1349].eqiad.wmnet

https://gerrit.wikimedia.org/r/1255689

gerritbot added a project: Patch-For-Review.Mar 19 2026, 11:55 AM
gerritbot added a comment.Mar 20 2026, 1:17 PM

Change #1255689 merged by JMeybohm:

[operations/puppet@production] wikikube: Add wikikube-worker[1335-1349].eqiad.wmnet

https://gerrit.wikimedia.org/r/1255689

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2026, 1:31 PM
JMeybohm closed this task as Resolved.Mar 20 2026, 2:29 PM
JMeybohm updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits