The .deb file is created with those repos:

• https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/28
• https://gitlab.wikimedia.org/repos/data-engineering/conda-base-env

I have checked with the Infrastructure-Foundations team that they are happy for us to use the .deb file created by GitLab-CI.
IRC log

So I have added the deb to apt.wikimedia.org with the following procedure:

• ssh to apt1001.wikimedia.org
• Download the artefact with: curl -o conda-base-env-0.0.1_amd64.deb https://gitlab.wikimedia.org/repos/data-engineering/conda-base-env/-/package_files/593/download
• Add it to our repo with: sudo -i reprepro -C main includedeb buster-wikimedia /home/btullis/conda-base/conda-base-env-0.0.1_amd64.deb

I have verified that it is now available for installation on hosts with:

btullis@an-test-client1001:~$ apt-cache policy conda-base-env
conda-base-env:
  Installed: (none)
  Candidate: 0.0.1
  Version table:
     0.0.1 1001
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages

Wow okay!

@Antoine_Quhen some Qs and Nits on the packaging:

1. package name. Maybe something other than conda-base-env would be more appropriate here. Perhaps conda-wmf? This isn't going to be a 'base' env for anything, in that we aren't going to stack it like we do for anaconda-wmf.

2. Do the linked spark executables work as is without activating the conda env? I found that doing conda_env/bin/spark-shell would fail finding the spark home, but e.g. conda_env/lib/python3.10/site-pacakges/pyspark/bin/spark-shell works.

BTW This is very exciting!

I was really hoping to get this finished today, but I'm still having issues.

Unpacking the conda packed tarball has an error

@Antoine_Quhen had a hack workaround to suppress this error, but I think it is really causing some issues. Something is wrong with the packed symlink fromn lib/python3.1 -> lib/python3.10. I don't know why this symlink exists in the first place.

conda-unpack is not doing everything it should

Notably, it is not rewriting the #!/usr/bin/env python headers in files like bin/ipython. This causes ipython not to work without activing the environment. I think this might be an artifact of the issues with the symlinks describe above, but I'm not sure.

To fix: I'm re-adding Antoine's --conda-pack-dest-prefix support to workflow_utils conda-dist. This will allow us to build .debs and specify conda-pack with the final install path, but it means the packed conda env won't be relocatable. That's okay for us I guess, as .debs hardcode their install paths, in this case to /opt/conda-analytics.

Dockerfile vs Gitlab CI

I've been using Antoine's really nice docker based build for development (faster than Gitlab CI for sure), but this means we really have two different build systems. I haven't synced any of my changes over to the Gitlab CI bits, so we'll have to work on that more.

I created a Draft MR with the stuff I'm trying:

https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/merge_requests/3

As of today, the changes have been ported between the Dockerfile and the Gitlab-ci.

So I am now working on the tar error and eventually the revert to --conda-pack-dest-prefix.

As of today, the changes have been ported between the Dockerfile and the Gitlab-ci.

Wow nice. Honestly the automated Gitlab-CI stuff is just SO COOL, but if it is difficult to maintain both, we can just use the Dockerfile build. It is def hard to develop this .deb using Gitlab-CI, eh?

@Antoine_Quhen something to think about, is if we will include the conda pkgs dir and files in this base conda env debian. These are the cached conda package files, and are what allow for a reliable 'clone' of the conda environment without having to retrieve a package from the internet.

Cloning using packages from the internet isn't so bad, but it seems that sometimes conda repos will remove a version of a package. When they do, cloning might fail, because the explicit version of a package in the base env is no longer available on the internet. (See this Slack thread too).

Adding pkgs will make the env much larger.

In the anaconda-wmf conda .deb, we solve this by creating 2 different .deb packages, one that has the base env, and other that depends on the base env, but includes the pkgs dir. We install the pkgs dir on stat boxes so users can clone, but only the conda env without pkgs on the analytics worker nodes, as there should be no reason why a env would be cloned there.

To see what I mean, compare the /usr/lib/anaconda-wmf env with /usr/lib/airflow on a stat box. /usr/lib/anaconda-wmf/pkgs exists and has a lot of stuff, but /usr/lib/airflow/pkgs does not.