Package request: install elixir and erlang-otp to the analytics clients
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	awight
	Tue, Apr 16, 3:18 PM

Description

My team has written a tool in elixir and we've become quite fond of it. After some experimentation with different execution contexts, it turns out that the analytics cluster is far and away the best place for us to run, mostly because we need fast access to the Enterprise HTML dumps which are mounted there. Cloud VPS disk access was at best 10% as fast.

In order to run our application, we're compiling cmake, erlang-otp and elixir manually for each user which is pretty much not ideal, but survivable. We would be happy if the following software could be installed by puppet:

Erlang/OTP 24, 25, or 26
Elixir 1.15 or 1.16
cmake

Compiling from source is easy and gives the safest binaries; I would be happy to help write or review deb packaging scripts. There are also precompiled packages available.

Event Timeline

awight created this task.Tue, Apr 16, 3:18 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Apr 16, 3:18 PM

awight updated the task description. (Show Details)Tue, Apr 16, 3:29 PM

Some of these packages already appear in debmonitor:

lbowmaker added a project: Data-Platform-SRE.Tue, Apr 16, 5:22 PM

It's worth nothing that the stat hosts are on Bullseye/Debian 11, which being provides the following versions:

Erlang 23.2.6
Elixir 1.10.3
cmake 3.18.4

Hi @awight - I'm happy to try to help here, but as @MoritzMuehlenhoff points out, trying to get packages from the Debian repositories that match your requirements may be quite tricky.
Some of our stat hosts (stat1004-8) are also still running buster, so their versions will be even further behind, although we're currently working to bring them up to date.

Have you thought about using conda environments for this requirement? See: https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Conda#Creating_a_new_environment for more information.
You will be able to install your own packages, principally from conda-forge and get more up-to-date packages in this way, without the need to build the toolchain from scratch for each user.

e.g.
https://anaconda.org/conda-forge/erlang (currently 26.2.4)
https://anaconda.org/conda-forge/elixir (currently 1.16.2)
https://anaconda.org/conda-forge/cmake (currently 3.29.2)

Will this approach be suitable for you?

@BTullis Thanks for highlighting this possibility! I tried the Conda environment as you suggested and it works perfectly for our needs. I was able to compile and run my app, and even at high concurrency the performance seems to be the same as with the bare metal environment I had cobbled together previously.

Closing the task :-)

Package request: install elixir and erlang-otp to the analytics clientsClosed, ResolvedPublicActions

Description

Event Timeline

Package request: install elixir and erlang-otp to the analytics clients
Closed, ResolvedPublic
Actions