Page MenuHomePhabricator
Create Task
Maniphest T414948

Decommission an-worker11[17-41] but reuse an-worker11[17,18,31,33,34] as dse-k8s-workers
Closed, ResolvedPublic
Actions

Description

There hosts have been replaced in {T383890} and can therefore be decommissioned.

We also need the space in the racks back.

Rename Hosts

  • an-worker1117.eqiad.wmnet -> dse-k8s-worker1024.eqiad.wmnet
  • an-worker1118.eqiad.wmnet -> dse-k8s-worker1025.eqiad.wmnet
  • an-worker1131.eqiad.wmnet -> dse-k8s-worker1026.eqiad.wmnet
  • an-worker1133.eqiad.wmnet -> dse-k8s-worker1027.eqiad.wmnet
  • an-worker1134.eqiad.wmnet -> dse-k8s-worker1028.eqiad.wmnet

Decommission Checklists

an-worker1119.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1120.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1121.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1122.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1123.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1124.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1125.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1126.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1127.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1128.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1129.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1130.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1132.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1135.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1136.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1137.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1138.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1139.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1140.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
an-worker1141.eqiad.wmnet

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove the site.pp definitions for decommissioned an-worker hostsoperations/puppetproduction+0 -4
Add the configuration for the new dse-k8s worker nodes that were an-workeroperations/puppetproduction+4 -4
Prepare to decom the old an-worker hostsoperations/puppetproduction+6 -53
Move a second journalnode to a newer hostoperations/puppetproduction+1 -1
Move an HDFS journalnode to a newer hostoperations/puppetproduction+1 -1
hadoop: decom an-worker1132operations/puppetproduction+0 -2
Exclude old an-worker hosts from HDFS and YARNoperations/puppetproduction+26 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.Jan 19 2026, 12:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 19 2026, 12:03 PM
BTullis renamed this task from Decommission an-worker11[17-41= to Decommission an-worker11[17-41].Jan 19 2026, 12:04 PM
BTullis claimed this task.
BTullis triaged this task as High priority.
BTullis added a project: Data-Platform-SRE (2026.01.05 - 2026.01.23).
BTullis updated the task description. (Show Details)
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.
gerritbot added a comment.Jan 19 2026, 12:43 PM

Change #1228479 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Exclude old an-worker hosts from HDFS and YARN

https://gerrit.wikimedia.org/r/1228479

gerritbot added a project: Patch-For-Review.Jan 19 2026, 12:43 PM
gerritbot added a comment.Jan 20 2026, 9:39 AM

Change #1228479 merged by Btullis:

[operations/puppet@production] Exclude old an-worker hosts from HDFS and YARN

https://gerrit.wikimedia.org/r/1228479

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2026, 10:32 AM
BTullis added a comment.Jan 20 2026, 10:56 AM

I've set these hosts into decommissioning mode, so they will be excluded from YARN and the HDFS blocks residing on them, will be replicate to other hosts.
We're now waiting for this graph to drop to zero, before decommissioning the servers.

image.png (1×1 px, 80 KB)

BTullis moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.Jan 21 2026, 2:51 PM

Down to 60 million under-replicated blocks.
https://grafana.wikimedia.org/goto/Wn7uN-Svg?orgId=1

image.png (1×1 px, 97 KB)

Gehel edited projects, added Data-Platform-SRE (2026.01.23 - 2026.02.13); removed Data-Platform-SRE (2026.01.05 - 2026.01.23).Jan 23 2026, 10:02 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.
BTullis added a comment.Feb 2 2026, 11:07 AM

We're down to approximately 20 million under-replicated files, now.

image.png (1×2 px, 216 KB)

Jclark-ctr mentioned this in T411919: hw troubleshooting: PERC1 battery failure for an-worker1148.Feb 10 2026, 2:09 PM
RKemper subscribed.Feb 10 2026, 4:30 PM

https://grafana.wikimedia.org/d/000000585/hadoop?orgId=1&from=2026-01-18T21:58:57.199Z&to=2026-02-16T21:40:15.121Z&timezone=utc&var-datasource=000000020&var-hadoop_cluster=analytics-hadoop&var-worker=$__all&viewPanel=panel-41

Under 7 million now. Should be 3-4 more days.

gerritbot added a comment.Wed, Feb 11, 9:05 PM

Change #1238826 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] hadoop: decom an-worker1132

https://gerrit.wikimedia.org/r/1238826

gerritbot added a project: Patch-For-Review.Wed, Feb 11, 9:05 PM
RKemper added a comment.Wed, Feb 11, 9:12 PM

Running decom cookbook, and have the puppet patch up. There's still changes to the private puppet repo and site.pp that will need to be made, but I think we're better off doing those all at once when we decom all the hosts. Just did the non-site.pp puppet patch stuff for now so I don't forget it later

ops-monitoring-bot added a comment.Wed, Feb 11, 9:12 PM

cookbooks.sre.hosts.decommission executed by ryankemper@cumin2002 for hosts: an-worker1132.eqiad.wmnet

  • an-worker1132.eqiad.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Host is already powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Host steps raised exception: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

BTullis added subscribers: wiki_willy, JAllemandou, RobH.Fri, Feb 13, 3:46 PM

@wiki_willy, @RobH

Sorry to be a pain, but would it be possible for us to retain around 5 of these hosts for a while, please?

It would be really helpful for us to be able to re-use them as Kubernetes workers with local storage to facilitate: T412925: Carry out end-user testing of spark on kubernetes
cc: @JAllemandou

We don't have a preference as to which hosts they would be, so we could be flexible around which rows/racks are under the least pressure to free up space.

wiki_willy added a comment.Fri, Feb 13, 6:00 PM

Sure @BTullis - no problem at all. Can you do me a favor though and let us know which hostnames on the EOL document, referencing this ticket number? That way, we'll leave it off of our radar. Thanks, Willy

In T414948#11615734, @BTullis wrote:

@wiki_willy, @RobH

Sorry to be a pain, but would it be possible for us to retain around 5 of these hosts for a while, please?

It would be really helpful for us to be able to re-use them as Kubernetes workers with local storage to facilitate: T412925: Carry out end-user testing of spark on kubernetes
cc: @JAllemandou

We don't have a preference as to which hosts they would be, so we could be flexible around which rows/racks are under the least pressure to free up space.

brouberol subscribed.Mon, Feb 16, 10:29 AM

@wiki_willy I've added comments on the EOL document, asking for

  • an-worker1117
  • an-worker1118
  • an-worker1131
  • an-worker1131
  • an-worker1134

to be retained.

Thank you!

brouberol added a comment.Mon, Feb 16, 10:32 AM

@BTullis FYI the rationale was that these hosts are on 4 different rows (A, B, C, D) and the 2 hosts in row C (1131 and 1134) are on two different racks (x and x).

Gehel edited projects, added Data-Platform-SRE (2026-02-13 - 2026-03-06); removed Data-Platform-SRE (2026.01.23 - 2026.02.13).Mon, Feb 16, 3:30 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
wiki_willy added a comment.Tue, Feb 17, 2:13 AM

Sounds good, thanks @brouberol !

gerritbot added a comment.Wed, Feb 18, 10:27 PM

Change #1238826 merged by Ryan Kemper:

[operations/puppet@production] hadoop: decom an-worker1132

https://gerrit.wikimedia.org/r/1238826

Maintenance_bot removed a project: Patch-For-Review.Wed, Feb 18, 10:30 PM
BTullis moved this task from Blocked/Waiting to In Progress on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.Mon, Feb 23, 2:55 PM
gerritbot added a comment.Mon, Feb 23, 10:23 PM

Change #1242508 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Move an HDFS journalnode to a newer host

https://gerrit.wikimedia.org/r/1242508

gerritbot added a project: Patch-For-Review.Mon, Feb 23, 10:23 PM
gerritbot added a comment.Mon, Feb 23, 10:32 PM

Change #1242511 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Move a second journalnode to a newer host

https://gerrit.wikimedia.org/r/1242511

gerritbot added a comment.Mon, Feb 23, 10:37 PM

Change #1242513 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Prepare to decom the old an-worker hosts

https://gerrit.wikimedia.org/r/1242513

gerritbot added a comment.Mon, Feb 23, 10:46 PM

Change #1242514 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add the configuration for the new dse-k8s worker nodes that were an-worker

https://gerrit.wikimedia.org/r/1242514

gerritbot added a comment.Mon, Feb 23, 10:49 PM

Change #1242516 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Remove the site.pp definitions for decommissioned an-worker hosts

https://gerrit.wikimedia.org/r/1242516

gerritbot added a comment.Tue, Feb 24, 9:58 AM

Change #1242508 merged by Btullis:

[operations/puppet@production] Move an HDFS journalnode to a newer host

https://gerrit.wikimedia.org/r/1242508

Stashbot added a comment.Tue, Feb 24, 12:35 PM

Mentioned in SAL (#wikimedia-analytics) [2026-02-24T12:35:39Z] <btullis> failing over HDFS namenode services to an-master1004 for T414948

gerritbot added a comment.Tue, Feb 24, 2:05 PM

Change #1242511 merged by Btullis:

[operations/puppet@production] Move a second journalnode to a newer host

https://gerrit.wikimedia.org/r/1242511

BTullis renamed this task from Decommission an-worker11[17-41] to Decommission an-worker11[17-41] but reuse an-worker11[17,18,31,33,34] as dse-k8s-workers.Tue, Feb 24, 2:33 PM
In T414948#11619648, @brouberol wrote:

@wiki_willy I've added comments on the EOL document, asking for

  • an-worker1117
  • an-worker1118
  • an-worker1131
  • an-worker1131
  • an-worker1134

to be retained.

Thank you!

I think that you meant an-worker1133 there, instead of a duplicate of 1131.

Stashbot added a comment.Tue, Feb 24, 2:38 PM

Mentioned in SAL (#wikimedia-analytics) [2026-02-24T14:38:52Z] <btullis> failing over HDFS namenode services to an-master1003 for T414948

gerritbot added a comment.Tue, Feb 24, 2:58 PM

Change #1242513 merged by Btullis:

[operations/puppet@production] Prepare to decom the old an-worker hosts

https://gerrit.wikimedia.org/r/1242513

gerritbot added a comment.Tue, Feb 24, 2:58 PM

Change #1242514 merged by Btullis:

[operations/puppet@production] Add the configuration for the new dse-k8s worker nodes that were an-worker

https://gerrit.wikimedia.org/r/1242514

BTullis updated the task description. (Show Details)Tue, Feb 24, 3:17 PM
BTullis updated the task description. (Show Details)Tue, Feb 24, 4:34 PM
ops-monitoring-bot added a comment.Tue, Feb 24, 4:52 PM

cookbooks.sre.hosts.decommission executed by btullis@cumin1003 for hosts: an-worker[1119-1130,1135-1141].eqiad.wmnet

  • an-worker1119.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1120.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1121.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1122.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1123.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1124.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1125.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1126.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1127.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1128.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1129.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1130.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1135.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1136.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1137.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1138.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1139.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1140.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1141.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
RobH unsubscribed.Tue, Feb 24, 5:01 PM
ops-monitoring-bot added a comment.Tue, Feb 24, 5:43 PM

Cookbook cookbooks.sre.hosts.rename started by btullis@cumin1003 from an-worker1117 to dse-k8s-worker1024 completed:

  • an-worker1117 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Tue, Feb 24, 6:37 PM

cookbooks.sre.hosts.decommission executed by btullis@cumin1003 for hosts: an-worker[1118,1131,1133-1134].eqiad.wmnet

  • an-worker1118.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1131.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1133.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
  • an-worker1134.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
ops-monitoring-bot added a comment.Tue, Feb 24, 6:53 PM

cookbooks.sre.hosts.decommission executed by btullis@cumin1003 for hosts: dse-k8s-worker1024.eqiad.wmnet

  • dse-k8s-worker1024.eqiad.wmnet (FAIL)
    • Host not found on Icinga, unable to downtime it
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB

ERROR: some step on some host failed, check the bolded items above

BTullis created subtask T418398: Two hosts are failing to do DHCP based PXE booting after renaming and moving vlan.Wed, Feb 25, 5:05 PM
BTullis created subtask T418582: Add dse-k8s-worker102[4-8] to the dse-k8s-eqiad cluster.Fri, Feb 27, 11:41 AM
BTullis removed BTullis as the assignee of this task.Fri, Feb 27, 11:46 AM
BTullis edited projects, added ops-eqiad; removed Patch-For-Review.
BTullis updated the task description. (Show Details)
Restricted Application added a project: DC-Ops. · View Herald TranscriptFri, Feb 27, 11:46 AM
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.Fri, Feb 27, 11:46 AM
Maintenance_bot added a project: SRE.Fri, Feb 27, 12:29 PM
Jclark-ctr claimed this task.Fri, Feb 27, 1:51 PM
Jclark-ctr moved this task from Backlog to High Priority Task on the ops-eqiad board.Fri, Feb 27, 1:59 PM
Jclark-ctr moved this task from High Priority Task to Decommission on the ops-eqiad board.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.Fri, Feb 27, 2:03 PM
Jclark-ctr added a comment.Fri, Feb 27, 3:07 PM

All cables have been removed from servers currently still in racks at this time will work on removing them thanks @BTullis

Jclark-ctr updated the task description. (Show Details)Wed, Mar 4, 1:52 PM
Jclark-ctr closed this task as Resolved.Wed, Mar 4, 1:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits