Page MenuHomePhabricator

Package versions in Conda-Analytics are not pinned
Closed, ResolvedPublic

Description

It might seem that versions of crucial packages in Conda-Analytics are pinned in conda-environment.yml (which in turn adds them to conda-environment.lock.yml).

However, those files just specify what versions should be installed in the new environment to start; Conda happily ignores them in all future transactions. This doesn't just mean that package A will be updated if the user runs conda update A. If the user runs conda install B and B lists A as a dependency, Conda will automatically upgrade A to the latest version (even if B's requirement is already satisfied by the existing version).

As you can imagine, this is a huge source of environment problems!

It should be easy to fix this by actually pinning versions when necessary by adding the specifications to a pinned file in the environment's conda-meta directory (docs).

Here's an example pinned file:

jupyter_core ==5.5.0
jupyter_server ==1.24.0
jupyter_telemetry ==0.1.0
jupyterhub ==1.5.0
jupyterhub-ldapauthenticator ==1.3.2
jupyterhub-singleuser ==1.5.0
jupyterhub-systemdspawner ==0.15.0
jupyterlab ==3.4.8
jupyterlab_pygments ==0.2.2
jupyterlab_server ==2.25.0
# https://phabricator.wikimedia.org/T356230
numpy <1.24
# https://phabricator.wikimedia.org/T356230
pandas <2.2
pyspark ==3.1.2
python ==3.10.*
sqlalchemy <2

Details

TitleReferenceAuthorSource BranchDest Branch
Update miniconda versionrepos/data-engineering/conda-analytics!47stevemuneneupdate_miniconda_24.4main
Clone the pinned file while creating a new cloned envrepos/data-engineering/conda-analytics!46stevemuneneclone_pinned_filemain
Pin essential conda-analytics packagesrepos/data-engineering/conda-analytics!43stevemunenepin_essential_conda_analytics_packagesmain
Draft: Pin essential conda-analytics packagesrepos/data-engineering/conda-analytics!42stevemunenepin_essential_conda_analytics_packagesmain
Customize query in GitLab

Event Timeline

Gehel triaged this task as Medium priority.Feb 9 2024, 1:29 PM
Gehel moved this task from Incoming to 2024.02.12 - 2024.03.03 on the Data-Platform-SRE board.

We have introduces a conda analytics pinned file with pandas and numpy versions for starters and built the dev deb package which we are going to test on an-test-client1002

Mentioned in SAL (#wikimedia-analytics) [2024-04-03T11:46:02Z] <stevemunene> disable puppet on an-test-client1002 to test new conda-analytics version T356231

New package introduces a pinned file for the base environment

stevemunene@an-test-client1002:~$ cat /opt/conda-analytics/conda-meta/pinned 
# https://phabricator.wikimedia.org/T356230
numpy <1.24
# https://phabricator.wikimedia.org/T356230
pandas <2.2

The current pinned file is a base model and we might need a more standardised production pinned file cc @nshahquinn-wmf

@Stevemunene I just created a new cloned Conda environment on an-test-clinet1002 using the Jupyter GUI. However, it doesn't have a pinned file:

nshahquinn-wmf@an-test-client1002:~/.conda/envs/2024-04-03T21.34.11_nshahquinn-wmf/conda-meta$ cat pinned
cat: pinned: No such file or directory

The current pinned file is a base model and we might need a more standardised production pinned file cc @nshahquinn-wmf

The "base model" pinned file does work pretty well: I manually added it to my environment, tried installing/updating a fairly long list of packages an analyst would likely use, and none of the non-pinned crucial packages (e.g. Python, Pyspark, Jupyter packages) got touched since they weren't in the dependency tree.

However, I still think it would be better to have a larger pinned file; in the past (before I introduced my own pinned files), I have broken things quite severely by trying to update Python or Jupyterlab or running conda update -all, and it's better to just make those safe even if most people aren't likely to try them 😁

Thanks @nshahquinn-wmf at the moment the pinned file can only be included in clones if the user wishes to. There is not yet a default way to avail this which does not have the optimal UX.
The pinned file can be availed during cloning by introducing the --pinned tag when cloning shown below;

stevemunene@an-test-client1002:~$ conda-analytics-clone test-pinned --pinned
Creating new cloned conda env test-pinned...
Source:      /opt/conda-analytics
Destination: /home/stevemunene/.conda/envs/test-pinned
.
.
.
Alternatively, you can use the conda-analytic helper script:
  source conda-analytics-activate test-pinned

Checking for the pinned file

stevemunene@an-test-client1002:~$ cat /opt/conda-analytics/conda-meta/pinned 
# https://phabricator.wikimedia.org/T356230
numpy <1.24
# https://phabricator.wikimedia.org/T356230
pandas <2.2

Looking for a way to go around this and avail it by default to all cloned environments for us.

We hit a bit of a delay with this while building the new conda package, so far we have updated the conda-analytics-clone command to include the --pinned tag so that the file is available to everyone using it as per Creating_a_new_environment.
However, we ran into a debian related challenge where the buster-backports repo is no longer available which prevents us from building any images that include his repo on apt-update failure. This is being tracked on T362518 and we plan to solve this by rebuilding our container on bullseye here T362648 so as to unblock progress on this.

We hit a bit of a delay with this while building the new conda package, so far we have updated the conda-analytics-clone command to include the --pinned tag so that the file is available to everyone using it as per Creating_a_new_environment.
However, we ran into a debian related challenge where the buster-backports repo is no longer available which prevents us from building any images that include his repo on apt-update failure. This is being tracked on T362518 and we plan to solve this by rebuilding our container on bullseye here T362648 so as to unblock progress on this.

This has been fixed after deploying v0.0.29 of conda-anaytics and the task is back in progress

Introduced a script to generate the conda pinned file based on the contents of the conda-environment.yml file that we use to generate the lock file. This provides a central place to manage the versions
https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/merge_requests/43

Doing a final build for the project with the --pinned tag but the tests seem to fail at the actual cloning step https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/blob/main/debian/bin/conda-analytics-clone#L28 Currently looking into this

Creating new cloned conda env mytestenv...
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda: error: unrecognized arguments: --pinned
The command '/bin/sh -c conda-analytics-clone mytestenv' returned a non-zero code: 2
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

Testing out the new package with the pinned file:
Downloading the package from build and installing

stevemunene@an-test-client1002:~$ sudo dpkg -i conda-analytics-0.0.31.dev_amd64.deb 
(Reading database ... 263137 files and directories currently installed.)
Preparing to unpack conda-analytics-0.0.31.dev_amd64.deb ...
Unpacking conda-analytics (0.0.30) over (0.0.29) ...
Setting up conda-analytics (0.0.30) ...
Post install script.
  Running /opt/conda-analytics/bin/python /opt/conda-analytics/bin/conda-unpack...

First we verify the initial location of the pinned file on the base env

stevemunene@an-test-client1002:~$ cat /opt/conda-analytics/conda-meta/pinned 
pyspark=3.1.2
numpy<1.24.0
pandas<2.0.0
pyarrow=9.0.0
jupyterhub=1.5.0
jupyterhub-systemdspawner=0.15.0
jupyterhub-ldapauthenticator=1.3.2
jupyterlab=3.4.8

Then we try the clone and verify the location on the cloned env

stevemunene@an-test-client1002:~$ conda-analytics-clone may24pinned
Creating new cloned conda env may24pinned...
Source:      /opt/conda-analytics
Destination: /home/stevemunene/.conda/envs/may24pinned
The following packages cannot be cloned out of the root environment:
 - conda-forge/linux-64::conda-23.10.0-py310hff52083_1
 - conda-forge/noarch::conda-libmamba-solver-23.12.0-pyhd8ed1ab_0
Packages: 223
Files: 1248

Downloading and Extracting Packages:


Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate may24pinned
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done

Next is verifying whether the pinned file is available in the cloned env via

stevemunene@an-test-client1002:~/.conda/envs/may24pinned/conda-meta$ find pinned
find: ‘pinned’: No such file or directory

The last couple of attempts were not successful, tying out new approaches to this.

It's worth noting that the --pinned option to conda create --clone is a requested feature and there is a patch that appears ready for merging into the upstream project.
https://github.com/conda/conda/pull/12499

The PR hadn't been touched since June 2023, so I added a comment yesterday and a contributor responded straight away, stating that they can look into completing the work.

So it might be possible to wait for this and upgrade to the latest version of conda, when it gets released, rather than spend too much time troubleshooting why this manual copy isn't working for us.

Hi @nshahquinn-wmf, the patch for this is ready and can be tested on an-testclient1002
We are looking to deploy this in the next conda-analytics version

With the new deb installed,

stevemunene@an-test-client1002:~$ conda-analytics-clone 0606mun
Creating new cloned conda env 0606mun...
Source:      /opt/conda-analytics
Destination: /home/stevemunene/.conda/envs/0606mun
.
.
.
Thu 06 Jun 2024 08:49:49 AM UTC Created user conda environment 0606mun

To activate this environment with vanilla conda run:
  source /opt/conda-analytics/etc/profile.d/conda.sh
  conda activate 0606mun

Alternatively, you can use the conda-analytic helper script:
  source conda-analytics-activate 0606mun

Successfully created an environment, checking for the pinned file verified and this should be ready to deploy

stevemunene@an-test-client1002:~$ cat /home/stevemunene/.conda/envs/0606mun/conda-meta/pinned 
pyspark=3.1.2
numpy<1.24.0
pandas<2.0.0
pyarrow=9.0.0
jupyterhub=1.5.0
jupyterhub-systemdspawner=0.15.0
jupyterhub-ldapauthenticator=1.3.2
jupyterlab=3.4.8

Having some issues with the deployment where the build process fails on gitlab CI but works everywhere else. The fail is not related to any of the work done for this ticket.
The build process failure is on the creation of the initial base image, and related to fetching some of the dependencies. currently investigating this

Step 30/73 : RUN source /opt/miniconda/bin/activate &&   conda env create --file conda-environment.lock.yml --name conda_dist_env
 ---> Running in baa8795e44b7
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): ...working... 
The command '/bin/bash -c source /opt/miniconda/bin/activate &&   conda env create --file conda-environment.lock.yml --name conda_dist_env' returned a non-zero code: 137
Cleaning up project directory and file based variables
00:02
ERROR: Job failed: exit code 1

However works on a local build

➜  conda-analytics git:(main) docker build --target conda_dist --platform linux/x86_64 -f docker/Dockerfile -t conda-analytics .

[+] Building 351.4s (20/20) FINISHED                                                                                                                                                                             docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                                             0.0s
 => => transferring dockerfile: 7.93kB                                                                                                                                                                                           0.0s
 => [internal] load metadata for docker-registry.wikimedia.org/bullseye:20240414                                                                                                                                                 1.2s
 => [internal] load .dockerignore                                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                                  0.0s
 => [conda_dist  1/15] FROM docker-registry.wikimedia.org/bullseye:20240414@sha256:a02dc8dbdee8ee00d5c8c98540f2e0f805815752ab7bb4f209ce58b4a0ae3a69                                                                              0.0s
 => [internal] load build context                                                                                                                                                                                                0.1s
 => => transferring context: 6.35kB                                                                                                                                                                                              0.1s
 => CACHED [conda_dist  2/15] RUN apt-get update &&   apt-get install -y curl                      ca-certificates                      git                       build-essential                      libsasl2-dev              0.0s
 => CACHED [conda_dist  3/15] RUN curl -o /tmp/minconda_installer.sh https://repo.anaconda.com/miniconda/Miniconda3-py310_23.10.0-1-Linux-x86_64.sh &&   chmod +x /tmp/minconda_installer.sh &&   echo "c7a34df472feb69805b64df  0.0s
 => CACHED [conda_dist  4/15] RUN mkdir -p /srv/conda-analytics                                                                                                                                                                  0.0s
 => CACHED [conda_dist  5/15] COPY pyproject.toml /srv/conda-analytics/                                                                                                                                                          0.0s
 => CACHED [conda_dist  6/15] COPY setup.cfg /srv/conda-analytics/                                                                                                                                                               0.0s
 => CACHED [conda_dist  7/15] COPY conda-environment.lock.yml /srv/conda-analytics/                                                                                                                                              0.0s
 => [conda_dist  8/15] COPY conda-pinned /srv/conda-analytics/                                                                                                                                                                   0.2s
 => [conda_dist  9/15] RUN /opt/miniconda/bin/pip install conda-pack@git+https://gitlab.wikimedia.org/repos/data-engineering/conda-pack.git@e3a8735ae9c7e286c97b6d6becac012ee9b869b7                                             9.5s
 => [conda_dist 10/15] WORKDIR /srv/conda-analytics                                                                                                                                                                              0.1s 
 => [conda_dist 11/15] RUN source /opt/miniconda/bin/activate &&   conda env create --file conda-environment.lock.yml --name conda_dist_env                                                                                    308.8s 
 => [conda_dist 12/15] RUN curl -L https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.1_2.12/1.2.1/iceberg-spark-runtime-3.1_2.12-1.2.1.jar -o /opt/miniconda/envs/conda_dist_env/lib/python3.10/site-pa  8.9s 
 => [conda_dist 13/15] RUN source /opt/miniconda/bin/activate &&   mkdir -p /srv/conda-analytics/dist &&   conda-pack     --name conda_dist_env     --output /srv/conda-analytics/dist/conda_dist_env.tgz     --compress-level   9.6s 
 => [conda_dist 14/15] RUN rm /opt/miniconda/pkgs/*.{conda,tar.bz2} &&     mv /opt/miniconda/pkgs /srv/conda-analytics/dist/pkgs                                                                                                 4.7s 
 => [conda_dist 15/15] RUN rm -r /opt/miniconda                                                                                                                                                                                  0.9s 
 => exporting to image                                                                                                                                                                                                           7.2s 
 => => exporting layers                                                                                                                                                                                                          7.2s 
 => => writing image sha256:6e644369acab3255a096af933831a514e98c8b88cebba040a6c12ecfeabfefc9                                                                                                                                     0.0s
 => => naming to docker.io/library/conda-analytics                                                                                                                                                                               0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/lnr9iwuw63jgjif9cx33hfumy

Looking to probably upgrade the actual miniconda version

Job succeeded after the update to miniconda 24.4.0 https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/jobs/286812

Releasing this to an-test-client1002 to test for any issues with the version then we can schedule a release on June 18 2024 1300UTC

Mentioned in SAL (#wikimedia-analytics) [2024-06-18T07:42:09Z] <stevemunene> update miniconda version on an-test-client T356231

Testing out the new version on an-test-client

stevemunene@an-test-client1002:~$ sudo dpkg -i conda-analytics-0.0.32_amd64.deb 
(Reading database ... 263160 files and directories currently installed.)
Preparing to unpack conda-analytics-0.0.32_amd64.deb ...
Unpacking conda-analytics (0.0.32) over (0.0.30) ...
Setting up conda-analytics (0.0.32) ...
Post install script.
  Running /opt/conda-analytics/bin/python /opt/conda-analytics/bin/conda-unpack...

Trying a cloned environment

stevemunene@an-test-client1002:~$ conda-analytics-clone mun1806
Creating new cloned conda env mun1806...
Source:      /opt/conda-analytics
Destination: /home/stevemunene/.conda/envs/mun1806
The following packages cannot be cloned out of the root environment:
 - conda-forge/linux-64::conda-23.10.0-py310hff52083_1
 - conda-forge/noarch::conda-libmamba-solver-23.12.0-pyhd8ed1ab_0
Packages: 223
Files: 1249

Downloading and Extracting Packages:


Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate mun1806
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/stevemunene/.conda/envs/mun1806

  added / updated specs:
    - conda-libmamba-solver=23.12.0
    - conda=23.10.0


The following NEW packages will be INSTALLED:

  conda              conda-forge/linux-64::conda-23.10.0-py310hff52083_1 
  conda-libmamba-so~ conda-forge/noarch::conda-libmamba-solver-23.12.0-pyhd8ed1ab_0 



Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Tue 18 Jun 2024 07:45:32 AM UTC Created user conda environment mun1806

To activate this environment with vanilla conda run:
  source /opt/conda-analytics/etc/profile.d/conda.sh
  conda activate mun1806

Alternatively, you can use the conda-analytic helper script:
  source conda-analytics-activate mun1806

Verifying the pinned file

stevemunene@an-test-client1002:~$ cat /home/stevemunene/.conda/envs/mun1806/conda-meta/pinned 
pyspark=3.1.2
numpy<1.24.0
pandas<2.0.0
pyarrow=9.0.0
jupyterhub=1.5.0
jupyterhub-systemdspawner=0.15.0
jupyterhub-ldapauthenticator=1.3.2
jupyterlab=3.4.8

All seems to work as expected so we can plan to deploy today.

spark tests from the quickstart guide

Slight block on the trigger release job. The thirdparty/conda package required by the runners seems to be unavailable on them. Of note, we are using the docker-registry.wikimedia.org/bullseye:20240414 image.

$ echo "deb http://apt.wikimedia.org/wikimedia buster-wikimedia thirdparty/conda" >> /etc/apt/sources.list.d/wikimedia.list
$ mkdir -p /usr/share/man/man1
$ apt update
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Get:1 http://mirrors.wikimedia.org/debian bullseye InRelease [116 kB]
Get:2 http://security.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://apt.wikimedia.org/wikimedia bullseye-wikimedia InRelease [123 kB]
Get:4 http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease [173 kB]
Get:5 http://mirrors.wikimedia.org/debian bullseye-updates InRelease [44.1 kB]
Get:6 http://mirrors.wikimedia.org/debian bullseye-backports InRelease [49.0 kB]
Get:7 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [275 kB]
Get:8 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages [69.1 kB]
Get:9 http://mirrors.wikimedia.org/debian bullseye/main amd64 Packages [8068 kB]
Get:10 http://mirrors.wikimedia.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
Get:11 http://mirrors.wikimedia.org/debian bullseye-backports/main amd64 Packages [406 kB]
Get:12 http://mirrors.wikimedia.org/debian bullseye-backports/contrib amd64 Packages [6164 B]
Fetched 9396 kB in 2s (5788 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
W: Skipping acquire of configured file 'thirdparty/conda/binary-amd64/Packages' as repository 'http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease' doesn't have the component 'thirdparty/conda' (component misspelt in sources.list?)2 packages can be upgraded. Run 'apt list --upgradable' to see them.
$ apt install -y curl gcc g++ gpg git make ca-certificates-java conda openjdk-11-jdk jq
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package conda
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Oh dear, sorry Steve. That is my fault. I thought that it wasn't used. I have recently removed it in T364550: Remove unused thirdparty/conda repository.
Should we revert it?

Oh yes, looking at codesearch, I can now see that this is used in workflow-utils and dumps: https://codesearch.wmcloud.org/search/?q=thirdparty%2Fconda&files=&excludeFiles=&repos=
Of course, when I did my search previously, it was before @brouberol had completed T366878: Not all data engineering gitlab projects are indexed in Codesearch, which is probably why I missed it.

OK, I'll prepare a revert.

Change #1047933 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Revert "Remove conda repository from reprepro configuration"

https://gerrit.wikimedia.org/r/1047933

Thanks for the spot and quick fix @BTullis, there is mention of a manual delete of some files though https://phabricator.wikimedia.org/T364550#9905899 not sure if that will affect much after the revert

Change #1047933 merged by Btullis:

[operations/puppet@production] Revert "Remove conda repository from reprepro configuration"

https://gerrit.wikimedia.org/r/1047933

Another slight block on the deb build that I am looking into.

Step 44/73 : COPY --from=conda_dist ${WORK_DIR}/dist/pkgs ${WORK_DIR}/pkgs
Error processing tar file(exit status 1): write /srv/conda-analytics/pkgs/cache/497deca9.json: no space left on device
Cleaning up project directory and file based variables
00:02
ERROR: Job failed: exit code 1

Mentioned in SAL (#wikimedia-analytics) [2024-06-26T10:23:19Z] <stevemunene> debdeploy conda-analytics 0.0.32 to test cluster T356231

For the upgrade, first is
Download the package to the apt server then verify

stevemunene@apt1002:~$  dpkg-deb --info conda-analytics-0.0.32_amd64.deb 
 new Debian package, version 2.0.
 size 1135322116 bytes: control archive=936 bytes.
     570 bytes,    13 lines      control              
      49 bytes,     1 lines      files                
     645 bytes,    19 lines   *  postinst             #!/usr/bin/env
 Package: conda-analytics
 Version: 0.0.32
 Architecture: amd64
 Maintainer: Aqu (WMF) <aquhen@wikimedia.org>
 Installed-Size: 4387608
 Depends: bash, default-jre-headless | java8-runtime-headless, libsasl2-2, libmariadb-dev
 Section: python
 Priority: optional
 Homepage: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics
 Description: conda packed environment with pyspark for WMF
  This package contains the standalone environment conda-analytics.
  This package is intended to be installed across the analytics cluster to provide Spark 3 and other
  libraries.

update the version in apt repo

stevemunene@apt1002:~$ sudo -i reprepro includedeb buster-wikimedia `pwd`/conda-analytics-0.0.32_amd64.deb
Exporting indices...
stevemunene@apt1002:~$ sudo -i reprepro includedeb bullseye-wikimedia `pwd`/conda-analytics-0.0.32_amd64.deb
Exporting indices...
Deleting files no longer referenced...
stevemunene@apt1002:~$ sudo -i reprepro ls conda-analytics
conda-analytics | 0.0.32 |   buster-wikimedia | amd64
conda-analytics | 0.0.32 | bullseye-wikimedia | amd64
stevemunene@apt1002:~$

generate the debdeploy spec

stevemunene@cumin1002:~$ generate-debdeploy-spec
Please enter the name of source package (e.g. openssl). type '' or 'quit' to abort
>conda-analytics
Enter an optional comment, e.g. a reference to a security advisory or a CVE ID mapping
>T356231

tool           -> The updated packages is an enduser tool, can be
                  rolled-out immediately.
daemon-direct  -> Daemons which are restarted during update, but which
                  do no affect existing users.
daemon-disrupt -> Daemons which are restarted during update, where the
                  users notice an impact. The update procedure is almost
                  identical, but displays additional warnings
library        -> After a library is updated, programs may need to be
                  restarted to fully effect the change. In addition
                  to libs, some applications may also fall under this rule,
                  e.g. when updating QEMU, you might need to restart VMs.

Please enter the update type:
>tool
Please enter the version of conda-analytics fixed in bookworm. Leave blank if no fix is available/required for bookworm.
>
Please enter the version of conda-analytics fixed in bullseye. Leave blank if no fix is available/required for bullseye.
>0.0.32
Please enter the version of conda-analytics fixed in buster. Leave blank if no fix is available/required for buster.
>0.0.32

Usually every upgrade only modifies existing package names. There are rare exceptions
e.g. if a rebase to a new upstream release is necessary.

Enter an optional comma-separated list of binary package names
which are being switched to a new name.
Leave blank to skip
>
Spec file created as 2024-06-26-conda-analytics.yaml

Deploying to the test cluster

stevemunene@cumin1002:~$ sudo debdeploy deploy -u 2024-06-26-conda-analytics.yaml -s hadoop-test
Rolling out conda-analytics:
Non-daemon update, no service restart needed

These hosts are already up-to-date:
  an-test-client1002.eqiad.wmnet (1 hosts)

conda-analytics was updated: 0.0.29 -> 0.0.32
  an-test-coord1001.eqiad.wmnet,an-test-
master[1001-1002].eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet (6
hosts)

The package to be updated isn't installed on these hosts:
  an-test-ui1001.eqiad.wmnet (1 hosts)

Mentioned in SAL (#wikimedia-analytics) [2024-06-26T11:12:10Z] <stevemunene> deploy conda-analytics v 0.0.32 to analytics hadoop worker hosts T356231 T356230

Mentioned in SAL (#wikimedia-analytics) [2024-06-26T11:22:15Z] <stevemunene> deploy conda-analytics v 0.0.32 to analytics hadoop coordinator hosts T356231 T356230

Mentioned in SAL (#wikimedia-analytics) [2024-06-26T11:33:06Z] <stevemunene> deploy conda-analytics v 0.0.32 to analytics stat hosts T356231 T356230

Mentioned in SAL (#wikimedia-analytics) [2024-06-26T11:47:41Z] <stevemunene> deploy conda-analytics v 0.0.32 to analytics airflow hosts T356231 T356230

Moving on to production environments

stevemunene@cumin1002:~$ sudo cumin A:hadoop-worker 'apt update'
106 hosts will be targeted:
an-worker[1078-1175].eqiad.wmnet,analytics[1070-1077].eqiad.wmnet

stevemunene@cumin1002:~$ sudo debdeploy deploy -u 2024-06-26-conda-analytics.yaml -s hadoop-worker
Rolling out conda-analytics:
Non-daemon update, no service restart needed

conda-analytics was updated: 0.0.29 -> 0.0.32
  an-worker[1078-1175].eqiad.wmnet,analytics[1070-1077].eqiad.wmnet
(106 hosts)
stevemunene@cumin1002:~$ sudo cumin A:hadoop-coordinator 'apt update'
2 hosts will be targeted:
an-coord[1003-1004].eqiad.wmnet

stevemunene@cumin1002:~$ sudo debdeploy deploy -u 2024-06-26-conda-analytics.yaml -s hadoop-coordinator
Rolling out conda-analytics:
Non-daemon update, no service restart needed

conda-analytics was updated: 0.0.29 -> 0.0.32
  an-coord[1003-1004].eqiad.wmnet (2 hosts)

Stat hosts

stevemunene@cumin1002:~$ sudo cumin A:stat 'apt update'
4 hosts will be targeted:
stat[1008-1011].eqiad.wmnet

stevemunene@cumin1002:~$ sudo debdeploy deploy -u 2024-06-26-conda-analytics.yaml -s stat 
Rolling out conda-analytics:
Non-daemon update, no service restart needed

conda-analytics was updated: 0.0.29 -> 0.0.32
  stat[1008-1011].eqiad.wmnet (4 hosts)

stevemunene@cumin1002:~$ sudo cumin A:stat 'systemctl restart jupyterhub-conda.service'
4 hosts will be targeted:
stat[1008-1011].eqiad.wmnet
OK to proceed on 4 hosts? Enter the number of affected hosts to confirm or "q" to quit: 4
===== NO OUTPUT =====                                                                                                                  
PASS |█████████████████████████████████████████████████████████████████████████████████████████| 100% (4/4) [02:00<00:00, 30.22s/hosts]
FAIL |                                                                                                 |   0% (0/4) [02:00<?, ?hosts/s]
100.0% (4/4) success ratio (>= 100.0% threshold) for command: 'systemctl restar...ub-conda.service'.
100.0% (4/4) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Airflow hosts

stevemunene@cumin1002:~$ sudo debdeploy deploy -u 2024-06-26-conda-analytics.yaml -s an-airflow
Rolling out conda-analytics:
Non-daemon update, no service restart needed

conda-analytics was updated: 0.0.29 -> 0.0.32
  an-airflow[1002,1004-1007].eqiad.wmnet,an-launcher1002.eqiad.wmnet
(6 hosts)

All hosts verified to be running version 0.0.32

image.png (374×776 px, 52 KB)

Moving this to the done column