Page MenuHomePhabricator
Create Task
Maniphest T412255

Q2:rack/setup/install mc1055-72
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of mc1055-72.

Hostname / Racking / Installation Details

This section should list the racking restrictions for these hosts. If they shouldn't share a rack/row with one another, or any existing hosts. This section should also list the other details listed below.

Hostnames: What are the hostnames, and have you updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions ? mc10[55-72]
Racking Proposal: Where should these systems be racked? Can they share with any existing systems or should they avoid any other systems sharing their rack or row? Use https://fault-tolerance.toolforge.org/map to optimize this placement. *3 per row, in different racks if possible.*
Networking Setup: # of Connections:1- Speed:10G. - VLAN:Private/Public/Other(Specify) :
OS Distro: Bookworm (default unless otherwise specified)
Boot Method: Legacy BIOS or UEFI. Please note UEFI must have partman updates applied in advance of setup and is currently in pilot program: https://wikitech.wikimedia.org/wiki/UEFI_Boot
Sub-team Technical Contact: Who should our on-sites contact with any questions involving system racking and setup? @effie

Each host should have its own setup checklist copied and pasted into the list below.

mc1055
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1056
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1057
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1058
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1059
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1060
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1061
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1062
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1063
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1064
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1065
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1066
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1067
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1068
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1069
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1070
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1071
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
mc1072
  • Receive in system on procurement task T405292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
mcrouter_wancache: replace mc1037 with mc1055Effie Mouzelioperations/puppetproduction+1 -1
site.pp: Add mc1055-72Clément Goubertoperations/puppetproduction+5 -0
partman: New mc nodes need UEFIClément Goubertoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jhancock.wm mentioned this in Unknown Object (Task).Dec 10 2025, 6:53 PM
Maintenance_bot added a project: SRE.Dec 10 2025, 7:29 PM
Clement_Goubert subscribed.Dec 22 2025, 4:19 PM

These are supermicro nodes and require UEFI. Patch underway.

gerritbot added a comment.Dec 22 2025, 4:25 PM

Change #1220373 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] partman: New mc nodes need UEFI

https://gerrit.wikimedia.org/r/1220373

gerritbot added a project: Patch-For-Review.Dec 22 2025, 4:25 PM
Clement_Goubert added a subscriber: jijiki.Dec 22 2025, 4:25 PM
Jclark-ctr subscribed.Jan 12 2026, 7:45 PM

@Clement_Goubert Could you add these to site.pp file additionally?

gerritbot added a comment.Jan 14 2026, 9:55 AM

Change #1226782 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] site.pp: Add mc1055-72

https://gerrit.wikimedia.org/r/1226782

gerritbot added a comment.Jan 20 2026, 8:55 AM

Change #1220373 merged by Clément Goubert:

[operations/puppet@production] partman: New mc nodes need UEFI

https://gerrit.wikimedia.org/r/1220373

gerritbot added a comment.Jan 20 2026, 8:55 AM

Change #1226782 merged by Clément Goubert:

[operations/puppet@production] site.pp: Add mc1055-72

https://gerrit.wikimedia.org/r/1226782

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2026, 9:33 AM
MoritzMuehlenhoff mentioned this in T398611: Migrate all memcached* clusters to nftables.Jan 21 2026, 7:59 AM
jijiki added a subtask: T398611: Migrate all memcached* clusters to nftables.Feb 4 2026, 10:52 AM
jijiki added a parent task: Unknown Object (Task).
MLechvien-WMF subscribed.Feb 9 2026, 2:28 PM

@Jclark-ctr Can I confirm what is the next step on this?

Jclark-ctr added a comment.Feb 9 2026, 2:35 PM

@MLechvien-WMF onsite we are waiting for the order to be completed and delivered. T405292 is ticket for procurement this is what we are waiting on.

jijiki added a project: User-jijiki.Feb 9 2026, 4:57 PM
MLechvien-WMF edited projects, added: ServiceOps, ServiceOps-Upgrades-Hardware; removed: SRE.Feb 9 2026, 5:01 PM
MLechvien-WMF moved this task from Inbox to Radar (Awareness) on the ServiceOps board.
Clement_Goubert created subtask T418263: mc10[55-72] implementation tracking.Feb 24 2026, 3:54 PM
Clement_Goubert changed the status of subtask T418263: mc10[55-72] implementation tracking from Open to Stalled.Feb 24 2026, 4:06 PM
jijiki added a comment.Mar 10 2026, 1:17 PM

@Jclark-ctr do we have any updates as to when the servers will be delivered?

MLechvien-WMF added a comment.EditedMar 24 2026, 9:57 AM

@Jclark-ctr @Jhancock.wm gentle follow-up on that?

Jclark-ctr added a comment.Mar 24 2026, 10:00 AM
In T412255#11742409, @MLechvien-WMF wrote:

@Jclark-ctr @Jhancock.wm gentle follow-up on that?

For shipping updates procurement ticket might be best see most recently comments

jijiki closed subtask T398611: Migrate all memcached* clusters to nftables as Resolved.Mar 27 2026, 11:21 AM
Jclark-ctr claimed this task.Apr 17 2026, 2:24 PM
Jclark-ctr updated the task description. (Show Details)
ops-monitoring-bot added a comment.Apr 17 2026, 7:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1060.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1055.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1057.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1059.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1064.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1061.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1062.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:28 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1060.eqiad.wmnet with OS bookworm completed:

  • mc1060 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172010_jclark_2519069_mc1060.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Apr 17 2026, 8:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1063.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1058.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1065.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1055.eqiad.wmnet with OS bookworm completed:

  • mc1055 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172020_jclark_2521369_mc1055.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Apr 17 2026, 8:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1066.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:43 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1064.eqiad.wmnet with OS bookworm completed:

  • mc1064 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172024_jclark_2521863_mc1064.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Apr 17 2026, 8:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1061.eqiad.wmnet with OS bookworm completed:

  • mc1061 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172028_jclark_2522319_mc1061.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 8:47 PM
Jclark-ctr updated the task description. (Show Details)
ops-monitoring-bot added a comment.Apr 17 2026, 8:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1062.eqiad.wmnet with OS bookworm completed:

  • mc1062 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172032_jclark_2522438_mc1062.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Apr 17 2026, 8:54 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1057.eqiad.wmnet with OS bookworm completed:

  • mc1057 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172036_jclark_2521509_mc1057.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 8:54 PM
Jclark-ctr updated the task description. (Show Details)
ops-monitoring-bot added a comment.Apr 17 2026, 8:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1067.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 8:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1068.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 9:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1059.eqiad.wmnet with OS bookworm completed:

  • mc1059 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172042_jclark_2521937_mc1059.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 9:01 PM
ops-monitoring-bot added a comment.Apr 17 2026, 9:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1069.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 9:02 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1058.eqiad.wmnet with OS bookworm completed:

  • mc1058 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172045_jclark_2526693_mc1058.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 9:02 PM
ops-monitoring-bot added a comment.Apr 17 2026, 9:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1070.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 9:07 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1065.eqiad.wmnet with OS bookworm executed with errors:

  • mc1065 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172050_jclark_2528676_mc1065.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console mc1065.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Apr 17 2026, 9:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1065.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 9:13 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1066.eqiad.wmnet with OS bookworm completed:

  • mc1066 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172054_jclark_2531585_mc1066.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 9:13 PM
ops-monitoring-bot added a comment.Apr 17 2026, 9:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1063.eqiad.wmnet with OS bookworm completed:

  • mc1063 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172059_jclark_2526379_mc1063.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 9:16 PM
Jclark-ctr updated the task description. (Show Details)
ops-monitoring-bot added a comment.Apr 17 2026, 9:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1071.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 9:24 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1056.eqiad.wmnet with OS bookworm executed with errors:

  • mc1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console mc1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Apr 17 2026, 9:34 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1068.eqiad.wmnet with OS bookworm completed:

  • mc1068 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172110_jclark_2544631_mc1068.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 9:35 PM
ops-monitoring-bot added a comment.Apr 17 2026, 9:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1069.eqiad.wmnet with OS bookworm completed:

  • mc1069 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172118_jclark_2547011_mc1069.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 9:36 PM
ops-monitoring-bot added a comment.Apr 17 2026, 9:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1065.eqiad.wmnet with OS bookworm completed:

  • mc1065 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172128_jclark_2551678_mc1065.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 17 2026, 10:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1071.eqiad.wmnet with OS bookworm completed:

  • mc1071 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172151_jclark_2555092_mc1071.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Apr 17 2026, 10:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1072.eqiad.wmnet with OS bookworm

Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 10:10 PM
ops-monitoring-bot added a comment.Apr 17 2026, 10:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 10:15 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1067.eqiad.wmnet with OS bookworm executed with errors:

  • mc1067 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console mc1067.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Apr 17 2026, 10:23 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1070.eqiad.wmnet with OS bookworm executed with errors:

  • mc1070 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console mc1070.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Apr 17 2026, 10:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1067.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 10:40 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1072.eqiad.wmnet with OS bookworm completed:

  • mc1072 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172224_jclark_2566810_mc1072.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 10:42 PM
ops-monitoring-bot added a comment.Apr 17 2026, 10:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1070.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 17 2026, 10:56 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1056.eqiad.wmnet with OS bookworm completed:

  • mc1056 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172239_jclark_2567155_mc1056.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 10:56 PM
ops-monitoring-bot added a comment.Apr 17 2026, 11:26 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1067.eqiad.wmnet with OS bookworm completed:

  • mc1067 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172308_jclark_2572570_mc1067.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Apr 17 2026, 11:26 PM
ops-monitoring-bot added a comment.Apr 17 2026, 11:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1070.eqiad.wmnet with OS bookworm completed:

  • mc1070 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604172331_jclark_2575711_mc1070.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Apr 17 2026, 11:55 PM
Jclark-ctr updated the task description. (Show Details)

@jijiki install is finished

jijiki added a comment.Apr 20 2026, 12:53 PM

Thank you!

MLechvien-WMF changed the status of subtask T418263: mc10[55-72] implementation tracking from Stalled to Open.Apr 28 2026, 8:06 AM
gerritbot added a comment.May 11 2026, 12:28 PM

Change #1285785 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] mcrouter_wancache: replace mc1037 with mc1055

https://gerrit.wikimedia.org/r/1285785

gerritbot added a project: Patch-For-Review.May 11 2026, 12:28 PM
jijiki changed the status of subtask T418263: mc10[55-72] implementation tracking from Open to In Progress.May 11 2026, 2:41 PM
gerritbot added a comment.May 12 2026, 10:15 AM

Change #1285785 merged by Effie Mouzeli:

[operations/puppet@production] mcrouter_wancache: replace mc1037 with mc1055

https://gerrit.wikimedia.org/r/1285785

Maintenance_bot removed a project: Patch-For-Review.May 12 2026, 10:30 AM
jijiki closed subtask T418263: mc10[55-72] implementation tracking as Resolved.May 19 2026, 9:27 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits