Page MenuHomePhabricator

Building wheels for cryptography package requires rust compiler
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/wikimedia-toolhub-pipeline-test/620/console

Building wheels for collected packages: cryptography
  Running setup.py bdist_wheel for cryptography: started
  Running setup.py bdist_wheel for cryptography: finished with status 'error'
[...snip...]
  generating cffi module 'build/temp.linux-x86_64-3.7/_openssl.c'
  running build_rust
[...snip...]
error: Can not find Rust compiler

https://cryptography.io/en/latest/installation.html#rust

Building cryptography requires having a working Rust toolchain. The current minimum supported Rust version is 1.45.0. This is newer than the Rust most package managers ship, so users will likely need to install with the instructions below.

Event Timeline

This seems to be caused by the Step 8/25 : RUN python3 "-m" "pip" "install" "-U" "poetry==1.1.4" step in building the container. Potentially adding an --only-binary cryptography argument or possibly the more generic --prefer-binary flag. Unfortunately this is in code generated by Blubber, so the fix won't be quick.

bd808 triaged this task as High priority.
bd808 moved this task from Backlog to In Progress on the Toolhub board.

This mess is going to break all of our CI, so its top priority. I'm not going to set it to UBN! mostly because that designation in my opinion should be reserved for emergencies in deployed software and services.

I have confirmed that adding either --only-binary cryptography or --prefer-binary to the Blubber generated pip install -U poetry==1.1.14 commands bypasses the rust compilation attempt locally. This requires manual edits to the Blubber generated Dockerfile, so we need to pick one and upstream it to Blubber. --prefer-binary feels a bit more future proof, but may cause concerns for anyone who trusts blind binary compilation in our CI/CD pipeline over retrieving built wheels from pypi.

Are the upstream wheels linking against the system openssl (and other security-sensitive libraries) or bundling it? If it's the latter then I think we should have the ability to build our own wheels (assuming that lets us link against system openssl...). At the very least we should check to make sure upstream has a plan of updating openssl/etc. if a critical security release comes out.

Long-term I think we probably will want to have a Rust compiler present in the build toolchain just like we have gcc/g++ because I think (and hope!) this trend will continue of porting important code from C/C++ to Rust. I'm happy to assist with that work, but it shouldn't block Toolhub.

A quick hack workaround for this would be to set PIP_ONLY_BINARY=cryptography in the environment before the pip install -U poetry==1.1.4 instruction. Unfortunately this cannot be done through Blubber as Blubber does not inject the user provided environment settings until PhasePrivilegeDropped which is after the PhasePrivileged stage where Blubber generates the poetry install instructions. It's like Blubber was built to stop me from making ugly hacks! ;)

Are the upstream wheels linking against the system openssl (and other security-sensitive libraries) or bundling it? If it's the latter then I think we should have the ability to build our own wheels (assuming that lets us link against system openssl...). At the very least we should check to make sure upstream has a plan of updating openssl/etc. if a critical security release comes out.

Long-term I think we probably will want to have a Rust compiler present in the build toolchain just like we have gcc/g++ because I think (and hope!) this trend will continue of porting important code from C/C++ to Rust. I'm happy to assist with that work, but it shouldn't block Toolhub.

I have no knowledge of linking vs bundling, but in the upstream docs it does say:

Note
If you are on RHEL/CentOS/Fedora/Debian/Ubuntu or another distribution derived from the preceding list, then you should upgrade pip (in a virtual environment!) and attempt to install cryptography again before trying to install the Rust toolchain. These platforms will receive a binary wheel and require no compiler if you have an updated pip!

I have no knowledge of linking vs bundling, but in the upstream docs it does say:

Note
If you are on RHEL/CentOS/Fedora/Debian/Ubuntu or another distribution derived from the preceding list, then you should upgrade pip (in a virtual environment!) and attempt to install cryptography again before trying to install the Rust toolchain. These platforms will receive a binary wheel and require no compiler if you have an updated pip!

[00:24]  <  legoktm> https://cryptography.io/en/latest/installation.html#static-wheels "Cryptography ships statically-linked wheels for macOS, Windows, and Linux (via manylinux). This allows compatible environments to use the most recent OpenSSL, regardless of what is shipped by default on those platforms." sigh

It appears that I am not the only person struggling to deal with this upstream change: https://github.com/pyca/cryptography/issues/5771

In a massive collision of coincidence, @Joe submitted https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/661915 to remove && pip3 install --upgrade pip \ from the base image that Toolhub has been using. Without this instruction we get pip 18.1 in the base image. With it we had pip 20.2.4 (due to the build date of the 0.0.2-s1 tagged image). Using the overridden pip requires the more modern invocation of python3 -m pip ... rather than using the /usr/bin/pip3 wrapper script from the system install.

It turns out that pip 20.2 is new enough to automagically install the wheel rather than trying and failing to build the cryptography library from source.

Change 663091 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] pipeline: Hack around cryptography rust build requirement

https://gerrit.wikimedia.org/r/663091

@bd808 sorry for the breakage, but I think it's kinda-inavoidable: the python3-build-* images are designed to be used to build wheels *for installing on premises*. That needs to use the pip version of the distro we're targeting or we get incompatible wheels. The fact that it upgraded pip was just a copypasta accident.

It's of course not your fault, there is little in the repository spelling it out, but if you are building python wheels for usage in a docker image, you should start from the python-devel image (and we can tweak it of course to fit your needs as well).

@bd808 sorry for the breakage, but I think it's kinda-inavoidable: the python3-build-* images are designed to be used to build wheels *for installing on premises*. That needs to use the pip version of the distro we're targeting or we get incompatible wheels. The fact that it upgraded pip was just a copypasta accident.

It's of course not your fault, there is little in the repository spelling it out, but if you are building python wheels for usage in a docker image, you should start from the python-devel image (and we can tweak it of course to fit your needs as well).

No worries @Joe! I only tagged you because I found that the timing of your change, which I fundamentally agree with I think, to be extremely coincidental to the cryptography packaging changes. The blubber.yaml for Toolhub has always had a "fixme" comment in it about the base image being a sketchy choice that needs to be revisited. It is probably time to open a task about that specifically and have a discussion with y'all in the SRE services group about what a "better" base image for a modern'ish Python container should look like.

Change 663091 merged by jenkins-bot:
[wikimedia/toolhub@main] pipeline: Hack around cryptography rust build requirement

https://gerrit.wikimedia.org/r/663091

bd808 lowered the priority of this task from High to Medium.Feb 10 2021, 8:58 PM

Lowering priority with the hack in place, but this needs a better solution.

Lowering priority with the hack in place, but this needs a better solution.

T274435: Find/create proper Python >=3.7 base image for Toolhub is the better fix. This may all come up again as we get closer to a prod deploy if we get asked by SRE or Security to stop using the upstream wheel and instead compile locally, but we can deal with that when it comes up.