We will need to install karapace to our VM for the datahub MVP, so we should ensure that it is debianized.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | BTullis | T299910 Data Catalog MVP | |||
Resolved | • razzi | T301562 Set up karapace instance for datahub | |||
Resolved | • razzi | T301565 Create debian package of karapace |
Event Timeline
Change 770605 had a related patch set uploaded (by Razzi; author: Razzi):
[operations/puppet@production] karapace: add karapace role
I have been working on this and there is a deb at deneb.codfw.wmnet:/home/razzi/karapace-temp/karapace_2.1.3-py3.7-0_amd64.deb. To build this deb, I used the gitlab.wikimedia.org:repos/data-engineering/workflow_utils conda approach which builds a .tar.gz and then used the same approach as gerrit.wikimedia.org:29418/operations/debs/airflow to build the deb.
To build the .tar.gz, I used the following commands:
(first go to karapace repository and checkout the version you'd like to build) karapace $ git fetch --tags karapace $ git checkout 2.1.3 (then add this file) karapace $ cat conda-environment.yaml dependencies: - python=3.7 (then I downloaded this file: curl https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/raw/main/workflow_utils/Dockerfile.conda-dist > /tmp/Dockerfile.conda-dist and modified it - see below) docker build -t workflow_utils:conda_dist -f /tmp/Dockerfile.conda-dist . docker run --mount type=bind,source=(pwd),target=/srv/project workflow_utils:conda_dist
Locally modified Dockerfile.conda-dist (the installation needed git, so we had to take out the FROM steps)
# Installs miniconda and workflow_utils into a conda env in the container. # This container can then be used to run workflow_utils commands like # conda-dist. FROM docker-registry.wikimedia.org/wikimedia-buster AS workflow_utils_install RUN apt-get update && apt-get install -y wget curl ca-certificates git # Download our install-miniconda-env bash script to bootstrap a working miniconda environment. # You can override the version of miniconda and python installed here by setting # MINICONDA_VERSION and PYTHON_VERSION env variables. # These must refer to an installer version available at https://repo.anaconda.com/miniconda/ # These versions ARE NOT the versions of conda or python that will be in your conda dist env. # Those should be controlled by your project in a conda environment.yml file. #ENV MINICONDA_VERSION=4.10.3 #ENV PYTHON_VERSION=3.7 # TODO: if we can put this built image somewhere, we don't need to download this at build time, # but can COPY it in from local FS. RUN wget https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/raw/main/bin/install-miniconda-env -O /srv/install-miniconda-env && chmod 755 /srv/install-miniconda-env # Installs by default to /srv/miniconda RUN /srv/install-miniconda-env # TODO: perhaps parameterize this requirement URL? # If we could just publish this docker image somewhere,then this should just be the working copy of the repo. RUN /srv/miniconda/bin/pip install git+https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils.git@main # A new layer that now only contains the workflow_utils conda env. # This conda env can be used to run workflow_utils related commands, # especially conda-dist for automating packing of other projects # into conda envs. # FROM docker-registry.wikimedia.org/wikimedia-buster AS workflow_utils_env # Copy /srv/conda_env from the previous stage. # In this way we keep only the final conda env # without any of the build step leftovers. # COPY --from=workflow_utils_install /srv/miniconda /srv/miniconda # workflow_utils conda-dist ENTRYPOINT. # Usage: # docker run --mount type=bind,source=$(pwd)/test_project,target=/srv/project workflow_utils:conda_dist # mount source should be the path to your project dir, which should be mounted at /srv/project # in the container. # FROM workflow_utils_env as workflow_utils_conda_dist # This is needed for conda-dist to find conda CLI. ENV CONDA_EXE=/srv/miniconda/condabin/conda SHELL ["/bin/bash", "-c"] WORKDIR "/srv/project" ENTRYPOINT /srv/miniconda/bin/conda-dist --dist-env-prefix=/srv/conda_dist_env --dist-env-dest=/srv/project/conda_dist_env.$(date +%Y-%m-%dT%H.%M.%S).tgz /srv/project
Finally to make the deb I followed the template from build_airflow_deb here starting with build_debian_tree since I already built the conda env:
Still todo: upload the .deb to apt.wikimedia.org and iterate on https://gerrit.wikimedia.org/r/c/operations/puppet/+/770605 to install the package and set up a systemd unit that runs the server process.
Deb is uploaded: https://apt.wikimedia.org/wikimedia/pool/main/k/karapace/
Code is also on the standard operations/deb gerrit repository: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/karapace/+/refs/heads/debian
Change 771419 had a related patch set uploaded (by Razzi; author: Razzi):
[operations/puppet@production] karapace: add karapace role
Change 770605 abandoned by Razzi:
[operations/puppet@production] karapace: add karapace role
Reason:
New patch at https://gerrit.wikimedia.org/r/c/operations/puppet/+/771419
It turns out the .deb, as I created it, overwrites the system python, which we don't want; this caused the following error:
Mar 22 17:36:05 karapace1001 prometheus-puppet-agent-stats[1504435]: import yaml Mar 22 17:36:05 karapace1001 prometheus-puppet-agent-stats[1504435]: ModuleNotFoundError: No module named 'yaml' Mar 22 17:36:05 karapace1001 systemd[1]: prometheus_puppet_agent_stats.service: Main process exited, code=exited, status=1/FAILURE Mar 22 17:36:05 karapace1001 systemd[1]: prometheus_puppet_agent_stats.service: Failed with result 'exit-code'.
What we want is to install the .deb python into /usr/lib/karapace.
(useful command: view the files that were installed by a deb)
dpkg-deb -c /var/cache/apt/archives/karapace_2.1.3-py3.7-1_amd64.deb
Change 772912 had a related patch set uploaded (by Razzi; author: Razzi):
[operations/puppet@production] karapace: use karapace included python; set hostname
Change 772912 merged by Razzi:
[operations/puppet@production] karapace: use karapace included python; set hostname
Change 773387 had a related patch set uploaded (by Razzi; author: Razzi):
[operations/puppet@production] karapace: remove Type=notify
Change 773387 merged by Razzi:
[operations/puppet@production] karapace: remove Type=notify