Page MenuHomePhabricator
Create Task
Maniphest T398438

Decommission or recommission all snapshot and dumpsdata servers
Closed, ResolvedPublic
Actions

Description

Now that the current generation of dumps is running from Airflow and Kubernetes, we can re-purpose the previous bare-metal servers.

The following two servers are EoL and should be decommissioned:

  • dumpsdata1003
  • snapshot1010

The remaining snapshot servers (https://wikitech.wikimedia.org/wiki/Dumps/Snapshot_hosts) will likely make good dse-k8s-worker nodes, since they have relatively powerful CPUs, although only 64 GB of RAM each.
The snapshot servers have been renamed and re-commissioned like this.

  • snapshot1011 -> dse-k8s-worker1015
  • snapshot1012 -> dse-k8s-worker1016
  • snapshot1013 -> dse-k8s-worker1017
  • snapshot1015 -> dse-k8s-worker1018
  • snapshot1016 -> dse-k8s-worker1019

The dumpsdata servers are to be renamed and recommissioned like this:

  • dumpsdata1004 -> an-worker1233
  • dumpsdata1005 -> an-worker1234
  • dumpsdata1006 -> an-worker1235
  • dumpsdata1007 -> an-worker1236

The dumpsdata servers (https://wikitech.wikimedia.org/wiki/Dumps/Dumpsdata_hosts) have lower-spec CPU and are 2U servers with 12 x 4TB disks each.
These are a little under-specced as either Hadoop or dse-k8s workers, but we might be able to find a role for them, or we could simply mark them as spare for DC-Ops to allocate accordingly.

Details

Other Assignee
brouberol
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Re-enable YARN and HDFS on an-worker123[3-6]Btullisoperations/puppetproduction+1 -5
Add 28 new hadoop workers to the analytics_hadoop clusterBtullisoperations/puppetproduction+29 -6
Add four new (renamed) an-worker nodes to the Hadoop clusterBtullisoperations/puppetproduction+7 -3
Temporarily exlude the 4 new hadoop workers to facilitate vlan changeBtullisoperations/puppetproduction+5 -1
Remove references to dumpsdata serversBtullisoperations/puppetproduction+0 -147
Fix the partman config for the newly renamed an-worker123[3-6]Btullisoperations/puppetproduction+1 -1
Add four hadoop workers from repurposed dumpsdata hostsBtullisoperations/puppetproduction+2 -2
site: assign dse_k8s::worker role to dse-k8s-worker1019Brouberoloperations/puppetproduction+1 -1
Provision dse-k8s-worker1019Brouberoloperations/puppetproduction+5 -3
Provision dse-k8s-worker1018Brouberoloperations/puppetproduction+4 -2
Provision dse-k8s-worker1017Brouberoloperations/puppetproduction+4 -2
Provision dse-k8s-worker1016Brouberoloperations/puppetproduction+4 -2
Provision dse-k8s-worker1015Brouberoloperations/puppetproduction+6 -2
Remove last references to snapshot serversBtullisoperations/puppetproduction+0 -9
Remove the netbox/5.65.10.in-addr.arpa zone from list of includesBtullisoperations/dnsmaster+0 -4
Add more dse-k8s-worker nodes to site.ppBtullisoperations/puppetproduction+2 -2
Remove all puppet code related to snapshot and dumpsdata hostsBtullisoperations/puppetproduction+2 -7 K
Complete cleanup of clouddumps serversBtullisoperations/puppetproduction+0 -59
Stop deploying mediawiki to deployment-snapshot05 in betaBtullisoperations/puppetproduction+0 -1
Begin cleanup up the clouddumps serversBtullisoperations/puppetproduction+2 -66
Set all snapshot and dumpsdata servers into the insetup roleBtullisoperations/puppetproduction+7 -38
Stop deploying mediawiki to snapshot hosts with scapBtullisoperations/puppetproduction+1 -6
Disable all dumps timers on snapshot hostsBtullisoperations/puppetproduction+19 -19
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
BTullis added a comment.Aug 6 2025, 3:49 PM

Reopening because we still haven't re-commisisoned the four remaining dumpsdata servers.
I've asked in T401299: Investigate whether we can add RAM to dumpsdata100[4-7] from any decommissioned hosts whether we can find some RAM to allocate so that they would be useful as Hadoop worker nodes.

VRiley-WMF changed the status of subtask T401299: Investigate whether we can add RAM to dumpsdata100[4-7] from any decommissioned hosts from Open to In Progress.Aug 19 2025, 7:30 PM
VRiley-WMF closed subtask T401299: Investigate whether we can add RAM to dumpsdata100[4-7] from any decommissioned hosts as Resolved.Aug 19 2025, 11:09 PM
BTullis edited projects, added: Data-Platform-SRE (2025.08.16 - 2025.09.05); removed: Data-Platform-SRE (2025.07.26 - 2025.08.15).Aug 20 2025, 10:31 AM
BTullis moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.
BTullis updated the task description. (Show Details)Sep 1 2025, 1:35 PM
gerritbot added a comment.Sep 2 2025, 1:01 PM

Change #1184068 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add four hadoop workers from repurposed dumpsdata hosts

https://gerrit.wikimedia.org/r/1184068

gerritbot added a project: Patch-For-Review.Sep 2 2025, 1:02 PM
gerritbot added a comment.Sep 2 2025, 1:10 PM

Change #1184070 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Remove references to dumpsdata servers

https://gerrit.wikimedia.org/r/1184070

gerritbot added a comment.Sep 2 2025, 1:12 PM

Change #1184068 merged by Btullis:

[operations/puppet@production] Add four hadoop workers from repurposed dumpsdata hosts

https://gerrit.wikimedia.org/r/1184068

ops-monitoring-bot added a comment.Sep 3 2025, 8:34 AM

Cookbook cookbooks.sre.hosts.rename started by btullis@cumin1003 from dumpsdata1004 to an-worker1233 completed:

  • dumpsdata1004 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 3 2025, 8:40 AM

Cookbook cookbooks.sre.hosts.rename started by btullis@cumin1003 from dumpsdata1005 to an-worker1234 completed:

  • dumpsdata1005 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 3 2025, 8:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1234.eqiad.wmnet with OS bullseye

gerritbot added a comment.Sep 3 2025, 9:47 AM

Change #1184489 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Fix the partman config for the newly renamed an-worker123[3-6]

https://gerrit.wikimedia.org/r/1184489

gerritbot added a comment.Sep 3 2025, 9:53 AM

Change #1184489 merged by Btullis:

[operations/puppet@production] Fix the partman config for the newly renamed an-worker123[3-6]

https://gerrit.wikimedia.org/r/1184489

ops-monitoring-bot added a comment.Sep 3 2025, 11:01 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1234.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1234 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1234.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 3 2025, 11:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1234.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 3 2025, 11:46 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1234.eqiad.wmnet with OS bullseye completed:

  • an-worker1234 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509031129_btullis_825303_an-worker1234.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 4 2025, 1:43 PM

Cookbook cookbooks.sre.hosts.rename started by btullis@cumin1003 from dumpsdata1006 to an-worker1235 completed:

  • dumpsdata1006 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 4 2025, 1:57 PM

Cookbook cookbooks.sre.hosts.rename started by btullis@cumin1003 from dumpsdata1007 to an-worker1236 completed:

  • dumpsdata1007 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 4 2025, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 4 2025, 3:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 4 2025, 5:27 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1235 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1235.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 4 2025, 5:28 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1236 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1236.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 5 2025, 8:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2025, 8:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2025, 9:40 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye completed:

  • an-worker1235 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509050923_btullis_1116200_an-worker1235.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 5 2025, 9:44 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye completed:

  • an-worker1236 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509050927_btullis_1116245_an-worker1236.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 5 2025, 11:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2025, 1:17 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1233 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1233.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 5 2025, 1:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye

Gehel edited projects, added: Data-Platform-SRE (2025.09.05 - 2025.09.26); removed: Data-Platform-SRE (2025.08.16 - 2025.09.05).Sep 5 2025, 2:23 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
gerritbot added a comment.Sep 8 2025, 10:22 AM

Change #1184070 merged by Btullis:

[operations/puppet@production] Remove references to dumpsdata servers

https://gerrit.wikimedia.org/r/1184070

ops-monitoring-bot added a comment.Sep 8 2025, 11:42 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1233 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1233.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 11:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 8 2025, 1:31 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1233 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1233.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 1:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 8 2025, 2:31 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye completed:

  • an-worker1233 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509081413_btullis_1622199_an-worker1233.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Sep 8 2025, 2:32 PM
BTullis moved this task from Blocked/Waiting to In Progress on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.Sep 9 2025, 9:03 AM
Stashbot added a comment.Sep 9 2025, 4:24 PM

Mentioned in SAL (#wikimedia-releng) [2025-09-09T16:24:12Z] <Krinkle> Remove unused deployment-snapshot- prefix Hiera file. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/0c6a99c4e12e9d1a55ce353e302d7dc891ae1177 - Follows-up decom of deployment-snapshot05 host. ref T398438, T360995

Stashbot mentioned this in T360995: Migrate Wikimedia production from PHP 8.1 to PHP 8.3.Sep 9 2025, 4:24 PM
BTullis added a comment.Sep 10 2025, 12:56 PM

I have created the kerberos credentials for the four new Hadoop workers.

btullis@krb1002:~$ sudo generate_keytabs.py --realm WIKIMEDIA hadoop_creds_T398438.txt 
HTTP/an-worker1233.eqiad.wmnet@WIKIMEDIA
hdfs/an-worker1233.eqiad.wmnet@WIKIMEDIA
yarn/an-worker1233.eqiad.wmnet@WIKIMEDIA
Entry for principal HTTP/an-worker1233.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1233.eqiad.wmnet/hadoop/HTTP.keytab.
Entry for principal hdfs/an-worker1233.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1233.eqiad.wmnet/hadoop/hdfs.keytab.
Entry for principal yarn/an-worker1233.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1233.eqiad.wmnet/hadoop/yarn.keytab.
HTTP/an-worker1234.eqiad.wmnet@WIKIMEDIA
hdfs/an-worker1234.eqiad.wmnet@WIKIMEDIA
yarn/an-worker1234.eqiad.wmnet@WIKIMEDIA
Entry for principal HTTP/an-worker1234.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1234.eqiad.wmnet/hadoop/HTTP.keytab.
Entry for principal hdfs/an-worker1234.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1234.eqiad.wmnet/hadoop/hdfs.keytab.
Entry for principal yarn/an-worker1234.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1234.eqiad.wmnet/hadoop/yarn.keytab.
HTTP/an-worker1235.eqiad.wmnet@WIKIMEDIA
hdfs/an-worker1235.eqiad.wmnet@WIKIMEDIA
yarn/an-worker1235.eqiad.wmnet@WIKIMEDIA
Entry for principal HTTP/an-worker1235.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1235.eqiad.wmnet/hadoop/HTTP.keytab.
Entry for principal hdfs/an-worker1235.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1235.eqiad.wmnet/hadoop/hdfs.keytab.
Entry for principal yarn/an-worker1235.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1235.eqiad.wmnet/hadoop/yarn.keytab.
HTTP/an-worker1236.eqiad.wmnet@WIKIMEDIA
hdfs/an-worker1236.eqiad.wmnet@WIKIMEDIA
yarn/an-worker1236.eqiad.wmnet@WIKIMEDIA
Entry for principal HTTP/an-worker1236.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1236.eqiad.wmnet/hadoop/HTTP.keytab.
Entry for principal hdfs/an-worker1236.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1236.eqiad.wmnet/hadoop/hdfs.keytab.
Entry for principal yarn/an-worker1236.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/an-worker1236.eqiad.wmnet/hadoop/yarn.keytab.
btullis@krb1002:~$
BTullis added a comment.Sep 10 2025, 1:24 PM

I created the new RAID0 logical volumes.

On the two older hosts, an-worker123[34], I used the older megacli command:

sudo  megacli -CfgEachDskRaid0 WB RA Direct CachedBadBBU -a0

On the two newer hosts, an-worker123[56],I used the newer perccli64 command:

sudo perccli64 /c0 add vd each r0 wb ra

Now I am able to initialize these with the sre.hadoop.init-hadoop-workers cookbook.

gerritbot added a comment.Sep 10 2025, 2:04 PM

Change #1186998 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add four new (renamed) an-worker nodes to the Hadoop cluster

https://gerrit.wikimedia.org/r/1186998

gerritbot added a project: Patch-For-Review.Sep 10 2025, 2:04 PM
gerritbot added a comment.Sep 10 2025, 2:18 PM

Change #1186998 merged by Btullis:

[operations/puppet@production] Add four new (renamed) an-worker nodes to the Hadoop cluster

https://gerrit.wikimedia.org/r/1186998

gerritbot added a comment.Sep 10 2025, 4:05 PM

Change #1187029 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Temporarily exlude the 4 new hadoop workers to facilitate vlan change

https://gerrit.wikimedia.org/r/1187029

gerritbot added a comment.Sep 10 2025, 5:13 PM

Change #1187029 merged by Btullis:

[operations/puppet@production] Temporarily exlude the 4 new hadoop workers to facilitate vlan change

https://gerrit.wikimedia.org/r/1187029

ops-monitoring-bot added a comment.Sep 10 2025, 5:53 PM

cookbooks.sre.hosts.decommission executed by btullis@cumin1003 for hosts: an-worker[1233-1236].eqiad.wmnet

  • an-worker1233.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • an-worker1234.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • an-worker1235.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • an-worker1236.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Sep 11 2025, 3:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 12 2025, 8:43 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1233.eqiad.wmnet with OS bullseye completed:

  • an-worker1233 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202509111544_btullis_2117294_an-worker1233.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509111639_btullis_2117294_an-worker1233.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Stashbot added a comment.Sep 12 2025, 8:43 AM

Mentioned in SAL (#wikimedia-operations) [2025-09-12T08:43:43Z] <btullis@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "T398438 - btullis@cumin1003"

Stashbot added a comment.Sep 12 2025, 8:43 AM

Mentioned in SAL (#wikimedia-operations) [2025-09-12T08:43:48Z] <btullis@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "T398438 - btullis@cumin1003"

ops-monitoring-bot added a comment.Sep 12 2025, 8:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1234.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 12 2025, 9:36 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1234.eqiad.wmnet with OS bullseye completed:

  • an-worker1234 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509120914_btullis_2225247_an-worker1234.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 12 2025, 9:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 12 2025, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1235 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1235.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 12 2025, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 12 2025, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1235 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1235.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 12 2025, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 12 2025, 10:43 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1235 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1235.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 12 2025, 10:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2025, 11:30 AM
ops-monitoring-bot added a comment.Sep 12 2025, 11:48 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1235 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1235.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 12 2025, 11:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 12 2025, 1:21 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1235 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1235.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 12 2025, 2:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 12 2025, 2:40 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1235 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1235.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 12 2025, 2:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 12 2025, 4:15 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1236 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1236.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Gehel edited projects, added: Data-Platform-SRE (2025.09.26 - 2025.10.17); removed: Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
ops-monitoring-bot added a comment.Sep 29 2025, 11:14 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 29 2025, 12:40 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1235 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1235.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 29 2025, 12:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 29 2025, 12:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 29 2025, 1:21 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1236 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1236.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 29 2025, 1:22 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1235.eqiad.wmnet with OS bullseye completed:

  • an-worker1235 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509291259_btullis_494219_an-worker1235.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 29 2025, 1:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 29 2025, 2:06 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1236 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1236.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 29 2025, 2:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 29 2025, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1236.eqiad.wmnet with OS bullseye completed:

  • an-worker1236 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509291424_btullis_505620_an-worker1236.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
gerritbot added a comment.Sep 29 2025, 9:24 PM

Change #1192239 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add 28 new hadoop workers to the analytics_hadoop cluster

https://gerrit.wikimedia.org/r/1192239

gerritbot added a project: Patch-For-Review.Sep 29 2025, 9:24 PM
BTullis moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Sep 29 2025, 9:43 PM
gerritbot added a comment.Sep 30 2025, 8:48 AM

Change #1192239 merged by Btullis:

[operations/puppet@production] Add 28 new hadoop workers to the analytics_hadoop cluster

https://gerrit.wikimedia.org/r/1192239

Maintenance_bot removed a project: Patch-For-Review.Sep 30 2025, 9:32 AM
BTullis closed this task as Resolved.Sep 30 2025, 9:59 AM
BTullis updated the task description. (Show Details)
BTullis moved this task from To Be Deployed to Done on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.

All four of the remaining dumpsdata servers have now been re-commissioned as Hadoop workers, so this ticket can now be resolved.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 3 2025, 7:33 AM
Gehel added a project: Essential-Work.Oct 7 2025, 8:51 AM
gerritbot added a comment.Oct 10 2025, 8:34 AM

Change #1195154 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Re-enable YARN and HDFS on an-worker123[3-6]

https://gerrit.wikimedia.org/r/1195154

gerritbot added a project: Patch-For-Review.Oct 10 2025, 8:34 AM
gerritbot added a comment.Oct 10 2025, 8:58 AM

Change #1195154 merged by Btullis:

[operations/puppet@production] Re-enable YARN and HDFS on an-worker123[3-6]

https://gerrit.wikimedia.org/r/1195154

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2025, 9:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits