stevemunene@an-test-client1002:~$ conda-analytics-clone bullseye-test
Creating new cloned conda env bullseye-test...
Source:      /opt/conda-analytics
Destination: /home/stevemunene/.conda/envs/bullseye-test
The following packages cannot be cloned out of the root environment:
 - conda-forge/linux-64::conda-23.10.0-py310hff52083_1
 - conda-forge/noarch::conda-libmamba-solver-23.12.0-pyhd8ed1ab_0
Packages: 223
Files: 1248
.
.
..
.
.
.
.
Wed 17 Apr 2024 07:43:56 AM UTC Created user conda environment bullseye-test

To activate this environment with vanilla conda run:
  source /opt/conda-analytics/etc/profile.d/conda.sh
  conda activate bullseye-test

Alternatively, you can use the conda-analytic helper script:
  source conda-analytics-activate bullseye-test

cd /home/stevemunene/.conda/envs/spark34t/lib/python3.10/site-packages/pyspark/jars

zip -r ~/artifacts/spark-3.4.1-assembly.zip .

hdfs dfs -mkdir -p /user/stevemunene/artifacts

hdfs dfs -copyFromLocal ~/artifacts/spark-3.4.1-assembly.zip /user/stevemunene/artifacts
hdfs dfs -chmod +r /user/stevemunene/artifacts/spark-3.4.1-assembly.zip

hdfs dfs -mkdir -p /user/stevemunene/artifacts

%env SPARK_HOME=/home/stevemunene/.conda/envs/spark34t/lib/python3.10/site-packages/pyspark
%env SPARK_CONF_DIR=/etc/spark3/conf

import wmfdata

spark = wmfdata.spark.create_custom_session(
    master='yarn',
    spark_config={
        "spark.shuffle.service.name": 'spark_shuffle_3_4',
        "spark.shuffle.service.port": '7339',
        "spark.yarn.archive": "hdfs:///user/stevemunene/artifacts/spark-3.4.1-assembly.zip",
        "spark.dynamicAllocation.maxExecutors": 128,
        ##
        # extras to make Iceberg work on 3.4.1:
        ##
        "spark.jars.packages": "org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.4.1",
        "spark.jars.ivySettings": "/etc/maven/ivysettings.xml",  # fix jar pulling
    }
)

stevemunene@apt1002:~$ dpkg-deb --info conda-analytics-0.0.29_amd64.deb 
 new Debian package, version 2.0.
 size 1129593128 bytes: control archive=940 bytes.
     570 bytes,    13 lines      control              
      49 bytes,     1 lines      files                
     645 bytes,    19 lines   *  postinst             #!/usr/bin/env
 Package: conda-analytics
 Version: 0.0.29
 Architecture: amd64
 Maintainer: Aqu (WMF) <aquhen@wikimedia.org>
 Installed-Size: 4361641
 Depends: bash, default-jre-headless | java8-runtime-headless, libsasl2-2, libmariadb-dev
 Section: python
 Priority: optional
 Homepage: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics
 Description: conda packed environment with pyspark for WMF
  This package contains the standalone environment conda-analytics.
  This package is intended to be installed across the analytics cluster to provide Spark 3 and other
  libraries.

stevemunene@apt1002:~$ sudo -i reprepro includedeb buster-wikimedia `pwd`/conda-analytics-0.0.29_amd64.deb
Exporting indices...
stevemunene@apt1002:~$ sudo -i reprepro includedeb bullseye-wikimedia `pwd`/conda-analytics-0.0.29_amd64.deb
Exporting indices...
Deleting files no longer referenced..
stevemunene@apt1002:~$ sudo -i reprepro ls conda-analytics
conda-analytics | 0.0.29 |   buster-wikimedia | amd64
conda-analytics | 0.0.29 | bullseye-wikimedia | amd64

stevemunene@cumin1002:~$ generate-debdeploy-spec
Please enter the name of source package (e.g. openssl). type '' or 'quit' to abort
>conda-analytics
Enter an optional comment, e.g. a reference to a security advisory or a CVE ID mapping
>T356231

tool           -> The updated packages is an enduser tool, can be
                  rolled-out immediately.
daemon-direct  -> Daemons which are restarted during update, but which
                  do no affect existing users.
daemon-disrupt -> Daemons which are restarted during update, where the
                  users notice an impact. The update procedure is almost
                  identical, but displays additional warnings
library        -> After a library is updated, programs may need to be
                  restarted to fully effect the change. In addition
                  to libs, some applications may also fall under this rule,
                  e.g. when updating QEMU, you might need to restart VMs.

Please enter the update type:
>tool
Please enter the version of conda-analytics fixed in bookworm. Leave blank if no fix is available/required for bookworm.
>
Please enter the version of conda-analytics fixed in bullseye. Leave blank if no fix is available/required for bullseye.
>0.0.29
Please enter the version of conda-analytics fixed in buster. Leave blank if no fix is available/required for buster.
>0.0.29

Usually every upgrade only modifies existing package names. There are rare exceptions
e.g. if a rebase to a new upstream release is necessary.

Enter an optional comma-separated list of binary package names
which are being switched to a new name.
Leave blank to skip
>
Spec file created as 2024-04-23-conda-analytics.yaml

stevemunene@cumin1002:~$ sudo cumin A:hadoop-test 'apt update'
8 hosts will be targeted:
an-test-client1002.eqiad.wmnet,an-test-coord1001.eqiad.wmnet,an-test-master[1001-1002].eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet

stevemunene@an-test-client1002:~$ apt-cache policy conda-analytics
conda-analytics:
  Installed: 0.0.28
  Candidate: 0.0.29
  Version table:
     0.0.29 1001
       1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages
 *** 0.0.28 100
        100 /var/lib/dpkg/status

stevemunene@cumin1002:~$ sudo debdeploy deploy -u 2024-04-23-conda-analytics.yaml -s hadoop-test
Rolling out conda-analytics:
Non-daemon update, no service restart needed

conda-analytics was updated: 0.0.28 -> 0.0.29
  an-test-client1002.eqiad.wmnet,an-test-coord1001.eqiad.wmnet,an-
test-master[1001-1002].eqiad.wmnet,an-test-
worker[1001-1003].eqiad.wmnet (7 hosts)

The package to be updated isn't installed on these hosts:
  an-test-ui1001.eqiad.wmnet (1 hosts)

stevemunene@an-test-client1002:~$ apt-cache policy conda-analytics
conda-analytics:
  Installed: 0.0.29
  Candidate: 0.0.29
  Version table:
 *** 0.0.29 1001
       1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status