A dependency from the hive and hive-hcatalog packages on python2.7
We were unable to build sqoop for bookworm, so we will need to ensure that this does not cause an issue.

Details

Other Assignee: brouberol

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
preseed: add grub-installer stanzas to an-test-worker hosts	Brouberol	operations/puppet	production	+5 -0
an-test-client1002: stop trying to install airflow	Brouberol	operations/puppet	production	+0 -14
Update the scap target for deployment	Btullis	analytics/refinery/scap	master	+1 -1
datahub-next: Update the MariaDB database to the refreshed host	Btullis	operations/deployment-charts	master	+3 -3
hadoop-test: Put the refreshed nodes into insetup mode prior to decom	Btullis	operations/puppet	production	+13 -14
hadoop-test: Update the database server used by hive and druid	Btullis	operations/puppet	production	+3 -3
Move analytics-test-hive and analytics-test-presto to new host	Btullis	operations/dns	master	+2 -2
hadoop-test: Replace an-test-master namenodes with refreshed hosts	Btullis	operations/deployment-charts	master	+28 -28
hadoop-test: Switch to using the new namenodes	Btullis	operations/puppet	production	+4 -16
hadoop-test: Allow for a second hive-metastore	Btullis	operations/puppet	production	+1 -1
hadoop-test: Fix typo in namenodes	Btullis	operations/puppet	production	+2 -2
hadoop-test: Temporarily bootstrap the new namenodes	Btullis	operations/puppet	production	+12 -0
Add component/libmysql-java to bookworm	Btullis	operations/puppet	production	+1 -0
Fix an issue with disabling the jobs on an-test-coord1002	Btullis	operations/puppet	production	+0 -2
Correct typo in hiera for an-test-coord1002	Btullis	operations/puppet	production	+1 -1
Disable systemd timers for refine jobs on an-test-coord1002	Btullis	operations/puppet	production	+6 -1
Switch new an-test-master and an-test-coord nodes into their role	Btullis	operations/puppet	production	+9 -35
Fix the dummy secrets for the new an-test hosts	Btullis	labs/private	master	+0 -0
Add dummy keytabs for new an-test nodes.	Btullis	labs/private	master	+0 -0

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Also build the spark-3.1-yarn-shuffler package	repos/data-engineering/spark!56	brouberol	T406746	main
	Allow the spark packages to be built for bookworm	repos/data-engineering/spark!55	brouberol	T406746	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		brouberol	T401692 EPIC: Migrate Data Platform SRE-owned hosts to Bookworm or later
Resolved		brouberol	T406746 Ensure that our Hadoop stack works on bookworm
Resolved	Request	BTullis	T433494 decommission an-test-coord1001.eqiad.wmnet
Resolved		VRiley-WMF	T433578 Physical decommission of an-test-coord1001
Resolved	Request	BTullis	T433495 decommission an-test-master100[1-2]
Resolved		VRiley-WMF	T433576 Physical uninstall of an-test-master100[1-2]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Following @BTullis build notes (<3) I was able to build a bookworm-compatible presto deb with python3 in the launcher.py shebang. As this diverges from upstream, I have to introduce a quilt patch changing the shebang.

commit 591179deaf46273b9261babf009ba2ac2d964750 (HEAD -> debian, origin/debian)
Author: Balthazar Rouberol <brouberol@wikimedia.org>
Date:   20 minutes ago

    Set the launcher shebang to python3 to enable bookworm compatibility

diff --git a/debian/changelog b/debian/changelog
index 3a3e7bc..559052f 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+presto (0.288.1-2) bookworm-wikimedia; urgency=medium
+
+  * Set shebang to python3 in launcher.py script to ensure bookworm compatibility
+
+ -- Balthazar Rouberol <brouberol@wikimedia.org>  Wed, 5 Aug 2024 10:15:41 +0100
+
 presto (0.288.1-1) bullseye-wikimedia; urgency=medium

   * Imported new upstream version 0.288.1
diff --git a/debian/patches/python3-shebang b/debian/patches/python3-shebang
new file mode 100644
index 0000000..1506c90
--- /dev/null
+++ b/debian/patches/python3-shebang
@@ -0,0 +1,17 @@
+Description: Set a python3 shebang in the launcher script
+ We do this to be able to install presto on Debian bookworm, which does not
+ have `python` installed, only `python3`.
+ .
+ presto (0.288.1-1) bullseye-wikimedia; urgency=medium
+ .
+   * Imported new upstream version 0.288.1
+Author: Balthazar Rouberol <brouberol@wikimedia.org>
+
+--- presto-0.288.1.orig/bin/launcher.py
++++ presto-0.288.1/bin/launcher.py
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+
+ import errno
+ import os
diff --git a/debian/patches/series b/debian/patches/series
new file mode 100644
index 0000000..87761d3
--- /dev/null
+++ b/debian/patches/series
@@ -0,0 +1 @@
+python3-shebang

brouberol@build2002:~/presto$ GIT_PBUILDER_AUTOCONF=no DIST=bookworm WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-builder=git-pbuilder --source-option="--include-removal"
gbp:info: Creating presto_0.288.1.orig.tar.gz from 'master'
...
dh_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use)
 dpkg-source --include-removal -b .
dpkg-source: info: using options from presto-0.288.1/debian/source/options: --extend-diff-ignore=^\.gitreview$
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building presto using existing ./presto_0.288.1.orig.tar.gz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: building presto in presto_0.288.1-2.debian.tar.xz
...

brouberol@build2002:~/presto$ ls /var/cache/pbuilder/result/bookworm-amd64/presto
presto_0.288.1-2_amd64.buildinfo  presto_0.288.1-2.dsc              presto-cli_0.288.1-2_all.deb
presto_0.288.1-2_amd64.changes    presto_0.288.1-2_source.changes   presto-server_0.288.1-2_all.deb
presto_0.288.1-2.debian.tar.xz    presto_0.288.1.orig.tar.gz

Can I take the opportunity to mention that a 2GB deb is insane?

brouberol@build2002:~/presto$ ls -alh presto-server_0.288.1-2_all.deb
-rw-r--r-- 1 brouberol wikidev 2.0G Aug  5 10:06 presto-server_0.288.1-2_all.deb

Now that I have, let's look at what's inside.

brouberol@build2002:~/presto$ ar vx presto-server_0.288.1-2_all.deb
x - debian-binary
x - control.tar.xz
x - data.tar.xz
brouberol@build2002:~/presto$ ls
bin		data.tar.xz  debian-binary  NOTICE  presto-server_0.288.1-2_all.deb
control.tar.xz	debian	     lib	    plugin  README.txt
brouberol@build2002:~/presto$ tar xvf data.tar.xz
./
./etc/
./etc/presto/
./etc/presto/catalog/
./etc/presto/config.properties
./etc/presto/jvm.config
...
brouberol@build2002:~/presto$ head ./usr/lib/presto/bin/launcher.py
#!/usr/bin/env python3

import errno
import os
import platform
import sys
import traceback

from fcntl import flock, LOCK_EX, LOCK_NB
from optparse import OptionParser

Success!

Let's now add it to our apt repo.

brouberol@apt1002:~$ rsync -v rsync://build2002.codfw.wmnet/pbuilder-result/bookworm-amd64/presto*0.288* .
presto-cli_0.288.1-2_all.deb
presto-server_0.288.1-2_all.deb
presto_0.288.1-2.debian.tar.xz
presto_0.288.1-2.dsc
presto_0.288.1-2_amd64.buildinfo
presto_0.288.1-2_amd64.changes
presto_0.288.1-2_source.changes
presto_0.288.1.orig.tar.gz

sent 176 bytes  received 4,251,824,247 bytes  70,278,089.64 bytes/sec
total size is 4,250,785,832  speedup is 1.00
brouberol@apt1002:~$ sudo -i reprepro -C main include bookworm-wikimedia $(pwd)/presto_0.288.1-2_amd64.changes
Exporting indices...
brouberol@apt1002:~$

Finally, let's see what happens if we upgrade on an-test-coord1002.

brouberol@an-test-coord1002:~$ sudo apt-get upgrade presto-cli presto-server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-image-6.1.0-50-amd64 python3-debconf
Use 'sudo apt autoremove' to remove them.
The following packages have been kept back:
  linux-image-amd64
The following packages will be upgraded:
  linux-libc-dev linux-perf openjdk-17-jre openjdk-17-jre-headless presto-cli presto-server
6 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 2,146 MB of archives.
After this operation, 105 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://apt.wikimedia.org/wikimedia bookworm-wikimedia/main amd64 presto-cli all 0.288.1-2 [14.2 MB]
Get:2 http://security.debian.org/debian-security bookworm-security/main amd64 linux-libc-dev amd64 6.1.180-1 [2,332 kB]
Get:3 http://security.debian.org/debian-security bookworm-security/main amd64 linux-perf amd64 6.1.180-1 [3,296 kB]
Get:4 http://apt.wikimedia.org/wikimedia bookworm-wikimedia/main amd64 presto-server all 0.288.1-2 [2,082 MB]
Get:5 http://security.debian.org/debian-security bookworm-security/main amd64 openjdk-17-jre amd64 17.0.20+8-1~deb12u1 [215 kB]
Get:6 http://security.debian.org/debian-security bookworm-security/main amd64 openjdk-17-jre-headless amd64 17.0.20+8-1~deb12u1 [43.9 MB]
Fetched 2,146 MB in 13s (160 MB/s)
INFO:debmonitor:Got 6 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server(s)
(Reading database ... 322453 files and directories currently installed.)
Preparing to unpack .../0-linux-libc-dev_6.1.180-1_amd64.deb ...
Unpacking linux-libc-dev:amd64 (6.1.180-1) over (6.1.177-1) ...
Preparing to unpack .../1-linux-perf_6.1.180-1_amd64.deb ...
Leaving 'diversion of /usr/bin/perf to /usr/bin/perf.wrapper by linux-perf'
Leaving 'diversion of /usr/share/bash-completion/completions/perf to /usr/share/bash-completion/completions/perf.wrapper by linux-perf'
Leaving 'diversion of /usr/share/man/man1/perf.1.gz to /usr/share/man/man1/perf.wrapper.1.gz by linux-perf'
Unpacking linux-perf (6.1.180-1) over (6.1.177-1) ...
Preparing to unpack .../2-openjdk-17-jre_17.0.20+8-1~deb12u1_amd64.deb ...
Unpacking openjdk-17-jre:amd64 (17.0.20+8-1~deb12u1) over (17.0.19+10-1~deb12u2) ...
Preparing to unpack .../3-openjdk-17-jre-headless_17.0.20+8-1~deb12u1_amd64.deb ...
Unpacking openjdk-17-jre-headless:amd64 (17.0.20+8-1~deb12u1) over (17.0.19+10-1~deb12u2) ...
Preparing to unpack .../4-presto-cli_0.288.1-2_all.deb ...
Unpacking presto-cli (0.288.1-2) over (0.288.1-1) ...
Preparing to unpack .../5-presto-server_0.288.1-2_all.deb ...
Unpacking presto-server (0.288.1-2) over (0.288.1-1) ...
Setting up linux-libc-dev:amd64 (6.1.180-1) ...
Setting up linux-perf (6.1.180-1) ...
Setting up presto-cli (0.288.1-2) ...
Setting up openjdk-17-jre-headless:amd64 (17.0.20+8-1~deb12u1) ...
Installing new version of config file /etc/java-17-openjdk/security/java.security ...
Installing new version of config file /etc/java-17-openjdk/security/public_suffix_list.dat ...
Setting up presto-server (0.288.1-2) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for mailcap (3.70+nmu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for ca-certificates-java (20230710~deb12u1) ...
done.
Setting up openjdk-17-jre:amd64 (17.0.20+8-1~deb12u1) ...
brouberol@an-test-coord1002:~$ dpkg -l | grep presto
ii  presto-cli                            0.288.1-2                            all          Presto CLI
ii  presto-server                         0.288.1-2                            all          Presto Server

Let's have a look at the presto server service and its associated launcher script.

brouberol@an-test-coord1002:~$ systemctl status presto-server.service
● presto-server.service - Presto Server Node
     Loaded: loaded (/lib/systemd/system/presto-server.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/presto-server.service.d
             └─presto-umask.conf
     Active: active (running) since Wed 2026-08-05 10:17:52 UTC; 1min 0s ago
   Main PID: 1643659 (java)
...
brouberol@an-test-coord1002:~$ head /usr/lib/presto/bin/launcher.py
#!/usr/bin/env python3

import errno
import os
import platform
import sys
import traceback

from fcntl import flock, LOCK_EX, LOCK_NB
from optparse import OptionParser

MA! MAA! I DID A THING!

According to https://phabricator.wikimedia.org/T378954, we already have hadoop built and distributed for bookworm. Confirmed here:

brouberol@apt1002:~$ sudo -i reprepro -C thirdparty/bigtop15 list bookworm-wikimedia | grep hadoop
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-client 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-conf-pseudo 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-doc 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-datanode 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-fuse 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-journalnode 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-namenode 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-secondarynamenode 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-hdfs-zkfc 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-httpfs 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-kms 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-mapreduce 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-mapreduce-historyserver 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn-nodemanager 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn-proxyserver 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn-resourcemanager 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|amd64: hadoop-yarn-timelineserver 2.10.2-deb12-2
bookworm-wikimedia|thirdparty/bigtop15|i386: hadoop-doc 2.10.2-deb12-2
brouberol@apt1002:~$

I'm going to reimage an-test-worker1003 and see what happens.

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-worker1003.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-worker1003.eqiad.wmnet with OS bookworm completed:

an-test-worker1003 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608051415_brouberol_976641_an-test-worker1003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

After a reimage, the only failing service is hadoop-yarn-nodemanager

brouberol@an-test-worker1003:~$ sudo systemctl | grep failed
● hadoop-yarn-nodemanager.service                                                          loaded failed failed    LSB: Hadoop nodemanager

2026-08-05 14:44:46,139 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed in state INITED; cause: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:167)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:319)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:468)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:878)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:946)
2026-08-05 14:44:46,143 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
org.apache.hadoop.service.ServiceStateException: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:319)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:468)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:878)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:946)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:167)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	... 8 more
2026-08-05 14:44:46,144 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
org.apache.hadoop.service.ServiceStateException: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:319)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:468)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:878)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:946)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:167)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	... 8 more
2026-08-05 14:44:46,146 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
2026-08-05 14:44:46,146 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
2026-08-05 14:44:46,146 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2026-08-05 14:44:46,147 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.service.ServiceStateException: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:319)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:468)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:878)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:946)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:167)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	... 8 more
2026-08-05 14:44:46,151 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at an-test-worker1003/10.64.53.21

@JAllemandou would you be able to debug w/ me?

bking mentioned this in T434142: Figure out what to do with an-worker1189.Aug 5 2026, 7:56 PM

Ok, I was able to find a difference between an-test-worker1001 (bullseye) and an-test-worker1003 (bookworm):

brouberol@an-test-worker1001:~$ sudo find /usr/ -name "*shuffle*.jar"
/usr/lib/hadoop-yarn/lib/spark-3.3-yarn-shuffle.jar
/usr/lib/hadoop-yarn/lib/spark-3.5-yarn-shuffle.jar
/usr/lib/hadoop-yarn/lib/spark-3.1-yarn-shuffle.jar
/usr/lib/hadoop-yarn/lib/spark-3.4-yarn-shuffle.jar
/usr/lib/spark-3.3-yarn-shuffle/spark-3.3.2-yarn-shuffle.jar
/usr/lib/hive/lib/spark-network-shuffle_2.11-2.4.5.jar
/usr/lib/spark-3.5-yarn-shuffle/spark-3.5.8-yarn-shuffle.jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.10.2.jar
/usr/lib/spark-3.1-yarn-shuffle/spark-3.1.2-yarn-shuffle.jar
/usr/lib/spark-3.4-yarn-shuffle/spark-3.4.1-yarn-shuffle.jar
/usr/lib/hadoop/client/hadoop-mapreduce-client-shuffle.jar
/usr/lib/hadoop/client/hadoop-mapreduce-client-shuffle-2.10.2.jar

brouberol@an-test-worker1003:~$ sudo find /usr/ -name "*shuffle*.jar"
/usr/lib/hive/lib/spark-network-shuffle_2.11-2.4.5.jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.10.2.jar
/usr/lib/hadoop/client/hadoop-mapreduce-client-shuffle.jar
/usr/lib/hadoop/client/hadoop-mapreduce-client-shuffle-2.10.2.jar

brouberol@an-test-worker1003:~$ dpkg -l | grep spark
ii  conda-analytics                       0.0.39                               amd64        conda packed environment with pyspark for WMF
ii  conda-analytics-next                  0.0.46                               amd64        conda packed environment with pyspark for WMF

brouberol@an-test-worker1001:~$ dpkg -l | grep spark
ii  conda-analytics                       0.0.39                             amd64        conda packed environment with pyspark for WMF
ii  conda-analytics-next                  0.0.46                             amd64        conda packed environment with pyspark for WMF
ii  spark-3.1-yarn-shuffle                3.1.2-3                            amd64        Spark 3.1 YARN Shuffle Service
ii  spark-3.3-yarn-shuffle                3.3.2-3                            amd64        Spark 3.3 YARN Shuffle Service
ii  spark-3.4-yarn-shuffle                3.4.1-3                            amd64        Spark 3.4 YARN Shuffle Service
ii  spark-3.5-yarn-shuffle                3.5.8-4                            amd64        Spark 3.5 YARN Shuffle Service

So, it would seem that our puppet code does not install the spark shuffler on Bookworm.

The core issue seems to be that Spark hasn't been packaged for Bookworm

brouberol@apt1002:~$ sudo -i reprepro -C thirdparty/bigtop15 list bookworm-wikimedia | grep spark
brouberol@apt1002:~$ sudo -i reprepro -C thirdparty/bigtop15 list bullseye-wikimedia | grep spark
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-core 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-datanucleus 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-external 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-history-server 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-master 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-python 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-sparkr 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-thriftserver 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-worker 2.4.5-deb11-2
bullseye-wikimedia|thirdparty/bigtop15|amd64: spark-yarn-shuffle 2.4.5-deb11-2

Confirmed by puppet

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install spark-3.1-yarn-shuffle' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package spark-3.1-yarn-shuffle
E: Couldn't find any package by glob 'spark-3.1-yarn-shuffle'
E: Couldn't find any package by regex 'spark-3.1-yarn-shuffle'
Error: /Stage[main]/Bigtop::Hadoop::Nodemanager/Package[spark-3.1-yarn-shuffle]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install spark-3.1-yarn-shuffle' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package spark-3.1-yarn-shuffle
E: Couldn't find any package by glob 'spark-3.1-yarn-shuffle'
E: Couldn't find any package by regex 'spark-3.1-yarn-shuffle'
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install spark-3.3-yarn-shuffle' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package spark-3.3-yarn-shuffle
E: Couldn't find any package by glob 'spark-3.3-yarn-shuffle'
E: Couldn't find any package by regex 'spark-3.3-yarn-shuffle'
Error: /Stage[main]/Bigtop::Hadoop::Nodemanager/Package[spark-3.3-yarn-shuffle]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install spark-3.3-yarn-shuffle' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package spark-3.3-yarn-shuffle
E: Couldn't find any package by glob 'spark-3.3-yarn-shuffle'
E: Couldn't find any package by regex 'spark-3.3-yarn-shuffle'
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install spark-3.4-yarn-shuffle' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package spark-3.4-yarn-shuffle
E: Couldn't find any package by glob 'spark-3.4-yarn-shuffle'
E: Couldn't find any package by regex 'spark-3.4-yarn-shuffle'
Error: /Stage[main]/Bigtop::Hadoop::Nodemanager/Package[spark-3.4-yarn-shuffle]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install spark-3.4-yarn-shuffle' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package spark-3.4-yarn-shuffle
E: Couldn't find any package by glob 'spark-3.4-yarn-shuffle'
E: Couldn't find any package by regex 'spark-3.4-yarn-shuffle'
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install spark-3.5-yarn-shuffle' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package spark-3.5-yarn-shuffle
E: Couldn't find any package by glob 'spark-3.5-yarn-shuffle'
E: Couldn't find any package by regex 'spark-3.5-yarn-shuffle'
Error: /Stage[main]/Bigtop::Hadoop::Nodemanager/Package[spark-3.5-yarn-shuffle]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install spark-3.5-yarn-shuffle' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package spark-3.5-yarn-shuffle
E: Couldn't find any package by glob 'spark-3.5-yarn-shuffle'
E: Couldn't find any package by regex 'spark-3.5-yarn-shuffle'
Error: Systemd start for hadoop-yarn-nodemanager failed!

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/55

Allow the spark packages to be built for bookworm

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/55

Allow the spark packages to be built for bookworm

I was able to build the packages, after having slightly tweaked our build tooling to allow it to build for bookworm

brouberol@build2002:~/spark$ ls /var/cache/pbuilder/result/bookworm-amd64/spark-3.*
/var/cache/pbuilder/result/bookworm-amd64/spark-3.3-yarn-shuffle_3.3.2-4_amd64.buildinfo
/var/cache/pbuilder/result/bookworm-amd64/spark-3.3-yarn-shuffle_3.3.2-4_amd64.changes
/var/cache/pbuilder/result/bookworm-amd64/spark-3.3-yarn-shuffle_3.3.2-4_amd64.deb
/var/cache/pbuilder/result/bookworm-amd64/spark-3.3-yarn-shuffle_3.3.2-4.dsc
/var/cache/pbuilder/result/bookworm-amd64/spark-3.3-yarn-shuffle_3.3.2-4_source.changes
/var/cache/pbuilder/result/bookworm-amd64/spark-3.3-yarn-shuffle_3.3.2-4.tar.gz
/var/cache/pbuilder/result/bookworm-amd64/spark-3.4-yarn-shuffle_3.4.1-4_amd64.buildinfo
/var/cache/pbuilder/result/bookworm-amd64/spark-3.4-yarn-shuffle_3.4.1-4_amd64.changes
/var/cache/pbuilder/result/bookworm-amd64/spark-3.4-yarn-shuffle_3.4.1-4_amd64.deb
/var/cache/pbuilder/result/bookworm-amd64/spark-3.4-yarn-shuffle_3.4.1-4.dsc
/var/cache/pbuilder/result/bookworm-amd64/spark-3.4-yarn-shuffle_3.4.1-4_source.changes
/var/cache/pbuilder/result/bookworm-amd64/spark-3.4-yarn-shuffle_3.4.1-4.tar.gz
/var/cache/pbuilder/result/bookworm-amd64/spark-3.5-yarn-shuffle_3.5.8-4_amd64.buildinfo
/var/cache/pbuilder/result/bookworm-amd64/spark-3.5-yarn-shuffle_3.5.8-4_amd64.changes
/var/cache/pbuilder/result/bookworm-amd64/spark-3.5-yarn-shuffle_3.5.8-4_amd64.deb
/var/cache/pbuilder/result/bookworm-amd64/spark-3.5-yarn-shuffle_3.5.8-4.dsc
/var/cache/pbuilder/result/bookworm-amd64/spark-3.5-yarn-shuffle_3.5.8-4_source.changes
/var/cache/pbuilder/result/bookworm-amd64/spark-3.5-yarn-shuffle_3.5.8-4.tar.gz

The packages were added to the apt repo

brouberol@apt1002:~$ sudo -i reprepro -C main list bookworm-wikimedia | grep yarn
bookworm-wikimedia|main|amd64: spark-3.3-yarn-shuffle 3.3.2-5
bookworm-wikimedia|main|amd64: spark-3.4-yarn-shuffle 3.4.1-5
bookworm-wikimedia|main|amd64: spark-3.5-yarn-shuffle 3.5.8-5
bookworm-wikimedia|main|source: spark-3.3-yarn-shuffle 3.3.2-5
bookworm-wikimedia|main|source: spark-3.4-yarn-shuffle 3.4.1-5
bookworm-wikimedia|main|source: spark-3.5-yarn-shuffle 3.5.8-5

brouberol@an-test-worker1003:~$ sudo apt-cache search spark-3.5-yarn-shuffle
spark-3.5-yarn-shuffle - Spark 3.5 YARN Shuffle Service

The shuffle packages should be installed by puppet. Crossing fingers.

Hmm, I think we'll be missing the spark-3.1-yarn-shuffle package, as we have

yarn_multi_spark_shuffler_versions:
  '3.1': 7337
  '3.3': 7338
  '3.4': 7339
  '3.5': 7340

@JAllemandou are we still relying on spark 3.1 at that point?

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/56

Also build the spark-3.1-yarn-shuffler package

Joseph said (on slack

We use spark-3.1 as a default for most of the jobs, so yes, we do

I'm going to build a spark-3.1-yarn-shuffle package.

brouberol@apt1002:~$ sudo -i reprepro -C main include bookworm-wikimedia $(pwd)/spark-3.1-yarn-shuffle_3.1.2-5_amd64.changes
Exporting indices...

brouberol@an-test-worker1003:~$ sudo apt-cache search spark-3.1-yarn-shuffle
spark-3.1-yarn-shuffle - Spark 3.1 YARN Shuffle Service

brouberol@an-test-worker1003:~$ sudo run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for an-test-worker1003.eqiad.wmnet
Info: Applying configuration version '(db3017ca45) Manuel Arostegui - db1178: Disable notifications'
Notice: /Stage[main]/Bigtop::Hadoop::Nodemanager/Package[spark-3.1-yarn-shuffle]/ensure: created
Notice: /Stage[main]/Bigtop::Hadoop::Nodemanager/Systemd::Service[hadoop-yarn-nodemanager]/Service[hadoop-yarn-nodemanager]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Bigtop::Hadoop::Nodemanager/Systemd::Service[hadoop-yarn-nodemanager]/Service[hadoop-yarn-nodemanager]: Unscheduling refresh on Service[hadoop-yarn-nodemanager]
Notice: Applied catalog in 64.97 seconds

MA! MAA! I DID ANOTHER THING!

brouberol updated Other Assignee, added: brouberol.Aug 6 2026, 12:09 PM

@JAllemandou checked that the node was registered in HDFS and YARN:

brouberol@an-test-master1003:~$ sudo -u hdfs kerberos-run-command hdfs hdfs dfsadmin -report
...
Name: 10.64.53.21:50010 (an-test-worker1003.eqiad.wmnet)
Hostname: an-test-worker1003.eqiad.wmnet
Rack: /eqiad/D/6
Decommission Status : Normal
Configured Capacity: 15794593501184 (14.37 TB)
DFS Used: 449852320124 (418.96 GB)
Non DFS Used: 7319611012 (6.82 GB)
DFS Remaining: 14541343092736 (13.23 TB)
DFS Used%: 2.85%
DFS Remaining%: 92.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Aug 06 12:16:16 UTC 2026
Last Block Report: Thu Aug 06 10:24:05 UTC 2026

brouberol@an-test-master1003:~$ sudo -u hdfs kerberos-run-command hdfs yarn node -list
26/08/06 12:17:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to an-test-master1004-eqiad-wmnet
Total Nodes:3
         Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
an-test-worker1003.eqiad.wmnet:8041	        RUNNING	an-test-worker1003.eqiad.wmnet:8042	                           0
an-test-worker1001.eqiad.wmnet:8041	        RUNNING	an-test-worker1001.eqiad.wmnet:8042	                           0
an-test-worker1002.eqiad.wmnet:8041	        RUNNING	an-test-worker1002.eqiad.wmnet:8042	                           0

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/56

Also build the spark-3.1-yarn-shuffler package

Change #1321932 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] an-test-client1002: stop trying to install airflow

https://gerrit.wikimedia.org/r/1321932

Change #1321932 merged by Brouberol:

[operations/puppet@production] an-test-client1002: stop trying to install airflow

https://gerrit.wikimedia.org/r/1321932

I'm now going to reimage an-test-client1002 to bookworm.

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-client1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-client1002.eqiad.wmnet with OS bookworm executed with errors:

an-test-client1002 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-client1002.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-client1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-client1002.eqiad.wmnet with OS bookworm completed:

an-test-client1002 (WARN)
- Downtimed on Icinga/Alertmanager
- Unable to disable Puppet, the host may have been unreachable
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608061323_brouberol_1226919_an-test-client1002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Debian GNU/Linux 12 auto-installed on Thu Aug 6 13:12:55 UTC 2026.
Last login: Thu Aug  6 13:40:38 2026 from 2620:0:861:1:208:80:154:7

You do not have a valid Kerberos ticket in the credential cache, remember to kinit.
brouberol@an-test-client1002:~$ sudo systemctl | grep failed
brouberol@an-test-client1002:~$

Reimaging mostly worked, except some error manifesting /etc/presto/config.properties. I re-ran puppet and everything worked fine:

brouberol@an-test-client1002:~$ sudo run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Warning: The current total number of facts: 2601 exceeds the number of facts limit: 2048
Info: Caching catalog for an-test-client1002.eqiad.wmnet
Info: Applying configuration version '(811682f20d) CDobbins - Revert "hieradata: Temporarily point eqiad PyBals at codfw etcd"'
Notice: /Stage[main]/Presto::Client/Presto::Properties[config]/File[/etc/presto/config.properties]/ensure: defined content as '{sha256}890c49d98a84134db09eb8587800ea2dc87cedfdc017c193df591d6b68e48eea'
Notice: Applied catalog in 54.08 seconds

I'll reimage both an-test-worker100[12] in the meantime

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-worker1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-worker1001.eqiad.wmnet with OS bookworm

After this, we will need to test upgrading the following hosts:

an-test-presto1001.eqiad.wmnet
an-test-ui1001.eqiad.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-ui1001.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-presto1001.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-worker1002.eqiad.wmnet with OS bookworm executed with errors:

an-test-worker1002 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-worker1002.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Hmm, somehow the an-test-worker100[12] hosts don't come back after a reboot. The debian installation probably fails?

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-worker1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-worker1002.eqiad.wmnet with OS bookworm executed with errors:

an-test-worker1002 (FAIL)
- Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-worker1002.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-worker1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-worker1001.eqiad.wmnet with OS bookworm executed with errors:

an-test-worker1001 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-worker1001.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-presto1001.eqiad.wmnet with OS bookworm completed:

an-test-presto1001 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608061434_brouberol_1236386_an-test-presto1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-ui1001.eqiad.wmnet with OS bookworm completed:

an-test-ui1001 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608061438_brouberol_1236201_an-test-ui1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-worker1001.eqiad.wmnet with OS bookworm

Change #1322002 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] preseed: add grub-installer stanzas to an-test-worker hosts

https://gerrit.wikimedia.org/r/1322002

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-worker1002.eqiad.wmnet with OS bookworm executed with errors:

an-test-worker1002 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-worker1002.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-worker1001.eqiad.wmnet with OS bookworm executed with errors:

an-test-worker1001 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-test-worker1001.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Change #1322002 merged by Brouberol:

[operations/puppet@production] preseed: add grub-installer stanzas to an-test-worker hosts

https://gerrit.wikimedia.org/r/1322002

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-worker1001.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-test-worker1002.eqiad.wmnet with OS bookworm

Seems that the preseed recipe was missing the following stanzas:

d-i	grub-installer/bootdev		string	/dev/sda /dev/sdb /dev/sdc /dev/sdd
# this workarounds LP #1012629 / Debian #666974
# it makes grub-installer to jump to step 2, where it uses bootdev
d-i	grub-installer/only_debian		boolean false

Thank you to @CDanis for figuring it out!

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-worker1001.eqiad.wmnet with OS bookworm completed:

an-test-worker1001 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608061605_brouberol_1249223_an-test-worker1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-test-worker1002.eqiad.wmnet with OS bookworm completed:

an-test-worker1002 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608061619_brouberol_1257067_an-test-worker1002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

brouberol@cumin1003:~$ sudo cumin 'an-test-*' 'grep CODENAME /etc/os-release'
10 hosts will be targeted:
an-test-client1002.eqiad.wmnet,an-test-coord1002.eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1003-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
OK to proceed on 10 hosts? Enter the number of affected hosts to confirm or "q" to quit: 10
===== NODE GROUP =====
(10) an-test-client1002.eqiad.wmnet,an-test-coord1002.eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1003-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'grep CODENAME /etc/os-release' -----
VERSION_CODENAME=bookworm

all an-test hosts now run on bookworm. I'm going to close this one.

brouberol closed this task as Resolved.Aug 6 2026, 5:40 PM

brouberol moved this task from In Progress to Done on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

Change #1322570 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] idm: sort airflow rules to help with next review

https://gerrit.wikimedia.org/r/1322570

Change #1322571 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] idm: add airflow-{analytics,analytics-product,platform-eng,wmde}-ops groups

https://gerrit.wikimedia.org/r/1322571

Change #1322572 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] admin: ensure all airflow-*-ops groups are marked as registered in bitu

https://gerrit.wikimedia.org/r/1322572

brouberol reopened this task as In Progress.Aug 7 2026, 7:26 AM

brouberol claimed this task.

brouberol moved this task from Done to Needs Review on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

brouberol closed this task as Resolved.Aug 7 2026, 7:54 AM

brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

^ these 3 patches were added by mistake

I'm re-opening this, as airflow-test-k8s.wikimedia.org / spark_submit_pi_yarn_cluster_mode / scheduled__2026-08-06T00:00:00+00:00 / spark_pi is now failing. As I understand from https://openillumi.com/en/en-pyspark-picklingerror-py311-spark331-fix/, the issue comes from the an-worker now having a python3.11 system installation, that is incompatible with pyspark 3.1.

The issue manifests itself via the following app logs:

Container: container_e456_1785345940263_4993_01_000001 on an-test-worker1002.eqiad.wmnet_8041_1786060858080
LogAggregationType: AGGREGATED
===========================================================================================================
LogType:stderr
LogLastModifiedTime:Fri Aug 07 00:00:58 +0000 2026
LogLength:19614
LogContents:
26/08/07 00:00:16 INFO SignalUtils: Registering signal handler for TERM
26/08/07 00:00:16 INFO SignalUtils: Registering signal handler for HUP
26/08/07 00:00:16 INFO SignalUtils: Registering signal handler for INT
26/08/07 00:00:17 INFO SecurityManager: Changing view acls to: analytics
26/08/07 00:00:17 INFO SecurityManager: Changing modify acls to: analytics
26/08/07 00:00:17 INFO SecurityManager: Changing view acls groups to:
26/08/07 00:00:17 INFO SecurityManager: Changing modify acls groups to:
26/08/07 00:00:17 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users  with view permissions: Set(analytics); groups with view permissions: Set(); users  with modify permissions: Set(analytics); groups with modify permissions: Set()
26/08/07 00:00:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
26/08/07 00:00:17 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1785345940263_4993_000001
26/08/07 00:00:17 INFO ApplicationMaster: Starting the user application in a separate Thread
26/08/07 00:00:17 INFO ApplicationMaster: Waiting for spark context initialization...
26/08/07 00:00:18 INFO SparkContext: Running Spark version 3.1.2
26/08/07 00:00:18 INFO ResourceUtils: ==============================================================
26/08/07 00:00:18 INFO ResourceUtils: No custom resources configured for spark.driver.
26/08/07 00:00:18 INFO ResourceUtils: ==============================================================
26/08/07 00:00:18 INFO SparkContext: Submitted application: PythonPi
26/08/07 00:00:18 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
26/08/07 00:00:18 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
26/08/07 00:00:18 INFO ResourceProfileManager: Added ResourceProfile id: 0
26/08/07 00:00:18 INFO SecurityManager: Changing view acls to: analytics
26/08/07 00:00:18 INFO SecurityManager: Changing modify acls to: analytics
26/08/07 00:00:18 INFO SecurityManager: Changing view acls groups to:
26/08/07 00:00:18 INFO SecurityManager: Changing modify acls groups to:
26/08/07 00:00:18 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users  with view permissions: Set(analytics); groups with view permissions: Set(); users  with modify permissions: Set(analytics); groups with modify permissions: Set()
26/08/07 00:00:18 INFO Utils: Successfully started service 'sparkDriver' on port 12000.
26/08/07 00:00:18 INFO SparkEnv: Registering MapOutputTracker
26/08/07 00:00:18 INFO SparkEnv: Registering BlockManagerMaster
26/08/07 00:00:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
26/08/07 00:00:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
26/08/07 00:00:18 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
26/08/07 00:00:18 INFO DiskBlockManager: Created local directory at /srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/blockmgr-69d25991-ade7-43ad-bdd3-85f6f0cb635f
26/08/07 00:00:18 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB
26/08/07 00:00:18 INFO SparkEnv: Registering OutputCommitCoordinator
26/08/07 00:00:18 INFO log: Logging initialized @2847ms to org.sparkproject.jetty.util.log.Slf4jLog
26/08/07 00:00:18 INFO Server: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_482-8u482-ga-1~deb12u1-b08
26/08/07 00:00:18 INFO Server: Started @2933ms
26/08/07 00:00:18 INFO AbstractConnector: Started ServerConnector@7ae4ca2e{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
26/08/07 00:00:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /jobs: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@55ff4ca2{/jobs,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /jobs/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@39b84c5{/jobs/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /jobs/job: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4e819a4b{/jobs/job,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /jobs/job/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@788e0b2c{/jobs/job/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /stages: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@475efef6{/stages,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /stages/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@451effeb{/stages/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /stages/stage: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@60acec3f{/stages/stage,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /stages/stage/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@988d65a{/stages/stage/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /stages/pool: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@56a57188{/stages/pool,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /stages/pool/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3e2e3faf{/stages/pool/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /storage: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@37b3ee72{/storage,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /storage/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7e6343ee{/storage/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /storage/rdd: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@325a6c92{/storage/rdd,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /storage/rdd/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2ba5ecd5{/storage/rdd/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /environment: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5f6f81ec{/environment,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /environment/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@739f872f{/environment/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /executors: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3fcdb83b{/executors,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /executors/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4da3853e{/executors/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /executors/threadDump: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2b1b4bbd{/executors/threadDump,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /executors/threadDump/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@266ba128{/executors/threadDump/json,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /static: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7f18b7e8{/static,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@53ac3177{/,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /api: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5b4434e1{/api,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /jobs/job/kill: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7e6ef7d7{/jobs/job/kill,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO ServerInfo: Adding filter to /stages/stage/kill: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4c7702d2{/stages/stage/kill,null,AVAILABLE,@Spark}
26/08/07 00:00:18 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://an-test-worker1002.eqiad.wmnet:4040
26/08/07 00:00:19 INFO YarnClusterScheduler: Created YarnClusterScheduler
26/08/07 00:00:19 INFO SparkHadoopUtil: Updating delegation tokens for current user.
26/08/07 00:00:19 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
26/08/07 00:00:19 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 13000.
26/08/07 00:00:19 INFO NettyBlockTransferService: Server created on an-test-worker1002.eqiad.wmnet:13000
26/08/07 00:00:19 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
26/08/07 00:00:19 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, an-test-worker1002.eqiad.wmnet, 13000, None)
26/08/07 00:00:19 INFO BlockManagerMasterEndpoint: Registering block manager an-test-worker1002.eqiad.wmnet:13000 with 366.3 MiB RAM, BlockManagerId(driver, an-test-worker1002.eqiad.wmnet, 13000, None)
26/08/07 00:00:19 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, an-test-worker1002.eqiad.wmnet, 13000, None)
26/08/07 00:00:19 INFO BlockManager: external shuffle service port = 7337
26/08/07 00:00:19 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, an-test-worker1002.eqiad.wmnet, 13000, None)
26/08/07 00:00:19 INFO ServerInfo: Adding filter to /metrics/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:19 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4b37ac25{/metrics/json,null,AVAILABLE,@Spark}
26/08/07 00:00:19 WARN Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
26/08/07 00:00:20 INFO SingleEventLogFileWriter: Logging events to hdfs:/var/log/spark/application_1785345940263_4993_1.lz4.inprogress
26/08/07 00:00:20 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
26/08/07 00:00:20 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
26/08/07 00:00:20 INFO YarnRMClient: Registering the ApplicationMaster
26/08/07 00:00:20 INFO ConfiguredRMFailoverProxyProvider: Failing over to an-test-master1004-eqiad-wmnet
26/08/07 00:00:20 INFO ApplicationMaster: Preparing Local resources
26/08/07 00:00:20 INFO ApplicationMaster:
===============================================================================
Default YARN executor launch context:
  env:
    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>{{PWD}}/__spark_conf__/__hadoop_conf__
    SPARK_YARN_STAGING_DIR -> hdfs://analytics-test-hadoop/user/analytics/.sparkStaging/application_1785345940263_4993
    SPARK_USER -> analytics
    PYTHONPATH -> /home/app/.local/lib/python3.11/site-packages:/opt/airflow/dags/airflow_dags:/opt/airflow<CPS>{{PWD}}/pyspark.zip<CPS>{{PWD}}/py4j-0.10.9-src.zip
    LD_LIBRARY_PATH -> /usr/lib/hadoop/lib/native
    REQUESTS_CA_BUNDLE -> /etc/ssl/certs/ca-certificates.crt

  command:
    {{JAVA_HOME}}/bin/java \
      -server \
      -Xmx1024m \
      '-Djava.net.useSystemProxies=True' \
      -Djava.io.tmpdir={{PWD}}/tmp \
      '-Dspark.port.maxRetries=100' \
      '-Dspark.network.crypto.keyLength=256' \
      '-Dspark.network.crypto.enabled=true' \
      '-Dspark.network.crypto.keyFactoryAlgorithm=PBKDF2WithHmacSHA256' \
      '-Dspark.driver.blockManager.port=13000' \
      '-Dspark.ui.port=4040' \
      '-Dspark.network.crypto.saslFallback=false' \
      '-Dspark.driver.port=12000' \
      '-Dspark.authenticate=true' \
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \
      -XX:OnOutOfMemoryError='kill %p' \
      org.apache.spark.executor.YarnCoarseGrainedExecutorBackend \
      --driver-url \
      spark://CoarseGrainedScheduler@an-test-worker1002.eqiad.wmnet:12000 \
      --executor-id \
      <executorId> \
      --hostname \
      <hostname> \
      --cores \
      1 \
      --app-id \
      application_1785345940263_4993 \
      --resourceProfileId \
      0 \
      --user-class-path \
      file:$PWD/__app__.jar \
      1><LOG_DIR>/stdout \
      2><LOG_DIR>/stderr

  resources:
    pyspark.zip -> resource { scheme: "hdfs" host: "analytics-test-hadoop" port: -1 file: "/user/analytics/.sparkStaging/application_1785345940263_4993/pyspark.zip" } size: 886596 timestamp: 1786060811676 type: FILE visibility: PRIVATE
    __spark_libs__ -> resource { scheme: "hdfs" host: "analytics-test-hadoop" port: -1 file: "/user/spark/share/lib/spark-3.1.2-assembly.jar" } size: 255798851 timestamp: 1686653031919 type: ARCHIVE visibility: PUBLIC
    py4j-0.10.9-src.zip -> resource { scheme: "hdfs" host: "analytics-test-hadoop" port: -1 file: "/user/analytics/.sparkStaging/application_1785345940263_4993/py4j-0.10.9-src.zip" } size: 41587 timestamp: 1786060811727 type: FILE visibility: PRIVATE
    __spark_conf__ -> resource { scheme: "hdfs" host: "analytics-test-hadoop" port: -1 file: "/user/analytics/.sparkStaging/application_1785345940263_4993/__spark_conf__.zip" } size: 164236 timestamp: 1786060811895 type: ARCHIVE visibility: PRIVATE

===============================================================================
26/08/07 00:00:20 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
26/08/07 00:00:20 INFO YarnAllocator: Resource profile 0 doesn't exist, adding it
26/08/07 00:00:20 INFO Configuration: resource-types.xml not found
26/08/07 00:00:20 INFO ResourceUtils: Unable to find 'resource-types.xml'.
26/08/07 00:00:20 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@an-test-worker1002.eqiad.wmnet:12000)
26/08/07 00:00:20 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
26/08/07 00:00:20 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
26/08/07 00:00:20 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done
26/08/07 00:00:20 INFO ServerInfo: Adding filter to /SQL: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:20 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4e136644{/SQL,null,AVAILABLE,@Spark}
26/08/07 00:00:20 INFO ServerInfo: Adding filter to /SQL/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:20 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5b09f2e1{/SQL/json,null,AVAILABLE,@Spark}
26/08/07 00:00:20 INFO ServerInfo: Adding filter to /SQL/execution: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:20 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3c9bf010{/SQL/execution,null,AVAILABLE,@Spark}
26/08/07 00:00:20 INFO ServerInfo: Adding filter to /SQL/execution/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:20 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1727f6f2{/SQL/execution/json,null,AVAILABLE,@Spark}
26/08/07 00:00:20 INFO ServerInfo: Adding filter to /static/sql: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
26/08/07 00:00:20 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3e8479e1{/static/sql,null,AVAILABLE,@Spark}
26/08/07 00:00:21 ERROR ApplicationMaster: User application exited with status 1
26/08/07 00:00:21 INFO ApplicationMaster: Final app status: FAILED, exitCode: 1, (reason: User application exited with status 1)
26/08/07 00:00:21 INFO SparkContext: Invoking stop() from shutdown hook
26/08/07 00:00:21 INFO AbstractConnector: Stopped Spark@7ae4ca2e{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
26/08/07 00:00:21 INFO SparkUI: Stopped Spark web UI at http://an-test-worker1002.eqiad.wmnet:4040
26/08/07 00:00:21 INFO YarnClusterSchedulerBackend: Shutting down all executors
26/08/07 00:00:21 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
26/08/07 00:00:21 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
26/08/07 00:00:21 INFO MemoryStore: MemoryStore cleared
26/08/07 00:00:21 INFO BlockManager: BlockManager stopped
26/08/07 00:00:21 INFO BlockManagerMaster: BlockManagerMaster stopped
26/08/07 00:00:21 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
26/08/07 00:00:21 INFO SparkContext: Successfully stopped SparkContext
26/08/07 00:00:21 INFO ShutdownHookManager: Shutdown hook called
26/08/07 00:00:21 INFO ShutdownHookManager: Deleting directory /srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/spark-8d358fde-108f-4745-ba23-f7e5493dca03
26/08/07 00:00:21 INFO ShutdownHookManager: Deleting directory /srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/spark-8d358fde-108f-4745-ba23-f7e5493dca03/pyspark-799dddee-e199-4cab-8765-af9783efe4a8

End of LogType:stderr
***********************************************************************

Container: container_e456_1785345940263_4993_01_000001 on an-test-worker1002.eqiad.wmnet_8041_1786060858080
LogAggregationType: AGGREGATED
===========================================================================================================
LogType:stdout
LogLastModifiedTime:Fri Aug 07 00:00:58 +0000 2026
LogLength:6435
LogContents:
Traceback (most recent call last):
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/serializers.py", line 437, in dumps
    return cloudpickle.dumps(obj, pickle_protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 72, in dumps
    cp.dump(obj)
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 540, in dump
    return Pickler.dump(self, obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 630, in reducer_override
    return self._function_reduce(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 503, in _function_reduce
    return self._dynamic_function_reduce(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 484, in _dynamic_function_reduce
    state = _function_getstate(func)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 156, in _function_getstate
    f_globals_ref = _extract_code_globals(func.__code__)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle.py", line 236, in _extract_code_globals
    out_names = {names[oparg] for _, oparg in _walk_global_ops(co)}
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle.py", line 236, in <setcomp>
    out_names = {names[oparg] for _, oparg in _walk_global_ops(co)}
                 ~~~~~^^^^^^^
IndexError: tuple index out of range
Traceback (most recent call last):
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/serializers.py", line 437, in dumps
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 72, in dumps
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 540, in dump
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 630, in reducer_override
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 503, in _function_reduce
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 484, in _dynamic_function_reduce
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle_fast.py", line 156, in _function_getstate
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle.py", line 236, in _extract_code_globals
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/cloudpickle/cloudpickle.py", line 236, in <setcomp>
IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pi.py", line 42, in <module>
    count = spark.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/rdd.py", line 997, in reduce
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/rdd.py", line 949, in collect
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/rdd.py", line 2949, in _jrdd
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/rdd.py", line 2828, in _wrap_function
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/rdd.py", line 2814, in _prepare_for_python_RDD
  File "/srv/hadoop/yarn/local/usercache/analytics/appcache/application_1785345940263_4993/container_e456_1785345940263_4993_01_000001/pyspark.zip/pyspark/serializers.py", line 447, in dumps
_pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range

End of LogType:stdout
***********************************************************************

I think that is shows that using a local master in cluster mode and spark < 3.4 is going to be a challenge when workers are running on Bookworn.

brouberol moved this task from Done to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Aug 7 2026, 10:27 AM

That might not be too much of an issue, as fixtures tend to show that only 3 DAGs in platform_eng would need to re-worked:

~/wmf/airflow-dags dev-makefile-improvements *26 ❯ rg "deploy-mode " -C 2 | grep 'cluster '                                                                                                                                                                                                                                                                                                                                                                                                   12:28:39 PM
tests/platform_eng/fixtures/spark_skein_specs/platform_eng_dags_differential_privacy_geoeditors.py-geoeditors_dag-do_dp_geoeditors_weekly.expected:    production --deploy-mode cluster venv/lib/python3.10/site-packages/differential_privacy/geoeditors_weekly.py
tests/platform_eng/fixtures/spark_skein_specs/platform_eng_dags_differential_privacy_country_project_page_daily.py-country_project_page_daily_dag-do_dp_pageview_actor.expected:    --queue production --deploy-mode cluster venv/lib/python3.10/site-packages/differential_privacy/country_project_page_gaussian.py
tests/platform_eng/fixtures/spark_skein_specs/platform_eng_dags_differential_privacy_geoeditors.py-geoeditors_dag-do_dp_geoeditors_monthly.expected:    --queue production --deploy-mode cluster venv/lib/python3.10/site-packages/differential_privacy/geoeditors_monthly.py

@JAllemandou would you mind checking my reasoning here?

Joseph and I spent a good hour trying to untangle the issue. TLDR we think that the only DAG that broke was spark_submit_pi_yarn_cluster_mode in test-k8s.

For context, that DAG was the first one I wrote when testing the integration of airflow-in-k8s and YARN. At the time, I wasn't really trying to model what we usually do in others DAGs, but mainly trying to test that airflow could talk to YARN, in whatever way.

This DAG runs a local spark-submit command:

spark-submit --master yarn --conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --name spark_pi --deploy-mode cluster /home/app/.local/lib/python3.11/site-packages/pyspark/examples/src/main/python/pi.py 10

When the an-test-worker100x host receives the pyspark.zip file (containing pyspark 3.1.2), and then uses its system python3.11 to execute it.

brouberol@an-test-worker1001:~/pyspark_unzip$ cat pyspark/version.py
__version__='3.1.2'

That breaks spectacularly. because pyspark 3.1.2 cannot be used with python3.1, due to waves computer issues. For that matter, any pyspark version prior to 3.4 will break.

However.

This is not a use case of ours. Indeed, we rely on skein to run the spark3-submit command on the hadoop worker itself, which relies on the conda-analytics environment, itself using python3.10.

brouberol@an-test-worker1001:~$ spark3-submit
Running /opt/conda-analytics/bin/spark-submit $@
SPARK_HOME: /opt/conda-analytics/lib/python3.10/site-packages/pyspark

We can see that using conda-analytics environment still works on bookworm

Password for brouberol@WIKIMEDIA:
brouberol@an-test-client1002:~$ spark3-submit --master yarn --deploy-mode cluster /opt/conda-analytics/lib/python3.10/site-packages/pyspark/examples/src/main/python/pi.py 10
Running /opt/conda-analytics/bin/spark-submit $@
SPARK_HOME: /opt/conda-analytics/lib/python3.10/site-packages/pyspark
Using Hadoop client lib jars at 3.2.0, provided by Spark.
...
26/08/07 12:56:29 INFO Client: Application report for application_1785345940263_5335 (state: RUNNING)
26/08/07 12:56:30 INFO Client: Application report for application_1785345940263_5335 (state: FINISHED)
26/08/07 12:56:30 INFO Client:
	 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
	 diagnostics: N/A
	 ApplicationMaster host: an-test-worker1002.eqiad.wmnet
	 ApplicationMaster RPC port: 12000
	 queue: default
	 start time: 1786107373712
	 final status: SUCCEEDED
	 tracking URL: http://an-test-master1004.eqiad.wmnet:8088/proxy/application_1785345940263_5335/
	 user: brouberol
26/08/07 12:56:30 INFO ShutdownHookManager: Shutdown hook called
26/08/07 12:56:30 INFO ShutdownHookManager: Deleting directory /tmp/spark-45bdb105-f0bc-40de-b06e-9b47dadf2720
26/08/07 12:56:30 INFO ShutdownHookManager: Deleting directory /tmp/spark-8d336e47-147e-447b-86b6-b0267bad2361

Given that all our spark-submit DAGs rely on skein, we should be fine.

I think the spark_submit_pi_yarn_cluster_mode test-k8s DAG is not particularly useful, as it's not testing anything "real" for us. I have paused it to avoid alerting anybody while I was out.

So, all that to say: @BTullis I'm not seeing any immediate blocker to upgrade our analytics hadoop cluster to Bookworm.

brouberol closed this task as Resolved.Aug 7 2026, 1:29 PM

brouberol moved this task from In Progress to Done on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

Gehel edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE (2026-07-03 - 2026-07-31).Aug 11 2026, 6:56 AM

Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.

BTullis closed subtask T433494: decommission an-test-coord1001.eqiad.wmnet as Resolved.Aug 11 2026, 8:25 AM

BTullis closed subtask T433495: decommission an-test-master100[1-2] as Resolved.

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-web1001.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-web1001.eqiad.wmnet with OS bookworm completed:

an-web1001 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608120804_btullis_2733470_an-web1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-presto1013.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-presto1013.eqiad.wmnet with OS bookworm executed with errors:

an-presto1013 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-presto1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-presto1013.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-presto1013.eqiad.wmnet with OS bookworm executed with errors:

an-presto1013 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-presto1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-presto1013.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-presto1013.eqiad.wmnet with OS bookworm completed:

an-presto1013 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608121023_btullis_2813895_an-presto1013.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-presto1015.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-presto1015.eqiad.wmnet with OS bookworm executed with errors:

an-presto1015 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- New OS is bullseye but bookworm was requested
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-presto1015.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-presto1015.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-presto1015.eqiad.wmnet with OS bookworm completed:

an-presto1015 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608121211_btullis_2900065_an-presto1015.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-presto1019.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-presto1019.eqiad.wmnet with OS bookworm completed:

an-presto1019 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608121356_btullis_2962644_an-presto1019.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.Aug 14 2026, 5:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-presto1010.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-presto1010.eqiad.wmnet with OS bookworm executed with errors:

an-presto1010 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-presto1010.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-presto1010.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-presto1010.eqiad.wmnet with OS bookworm executed with errors:

an-presto1010 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-presto1010.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1003 for host an-presto1010.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1003 for host an-presto1010.eqiad.wmnet with OS bookworm completed:

an-presto1010 (WARN)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608261321_brouberol_486125_an-presto1010.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Ensure that our Hadoop stack works on bookwormClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Ensure that our Hadoop stack works on bookworm
Closed, ResolvedPublic
Actions

Related Objects
Search...