Maniphest T204919

tox-docker fails on keyholder due to pycrypto
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Sep 20 2018, 8:43 AM

Description

https://gerrit.wikimedia.org/r/#/c/operations/software/keyholder/+/458236/ introduces dependencies on pynacl and pytcrypto. CI runs the tests using docker-registry.wikimedia.org/releng/tox:0.1.1 which seems to be missing development libraries.

Building wheels for collected packages: construct, pycrypto, pycparser
  Running setup.py bdist_wheel for pycrypto: started
  Running setup.py bdist_wheel for pycrypto: finished with status 'error'
...
  checking whether we are cross compiling... configure: error: in `/tmp/pip-install-9a29i208/pycrypto':
  configure: error: cannot run C compiled programs.
  If you meant to cross compile, use `--host'.
  See `config.log' for more details

    File "/tmp/pip-install-9a29i208/pycrypto/setup.py", line 278, in run
      raise RuntimeError("autoconf error")
  RuntimeError: autoconf error

Related Objects

Mentioned In: T203181: Quibble MariaDB should use a tmpfs as a datadir
Mentioned Here: T203183: Measuring TemplateWizard's impact on template usage
T203181: Quibble MariaDB should use a tmpfs as a datadir

Event Timeline

hashar created this task.Sep 20 2018, 8:43 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2018, 8:43 AM

At least the pycrypto install works:

$ docker pull docker-registry.wikimedia.org/releng/tox:0.1.1
$ docker run --rm -it --entrypoint=/bin/bash docker-registry.wikimedia.org/releng/tox:0.1.1
$ pip3 install --target . pycrypto

Collecting pycrypto
  Downloading https://files.pythonhosted.org/packages/60/db/645aa9af249f059cc3a368b118de33889219e0362141e75d4eaf6f80f163/pycrypto-2.6.1.tar.gz (446kB)

Building wheels for collected packages: pycrypto
  Running setup.py bdist_wheel for pycrypto ... done
  Stored in directory: /cache/pip/wheels/27/02/5e/77a69d0c16bb63c6ed32f5386f33a2809c94bd5414a2f6c196
Successfully built pycrypto
Installing collected packages: pycrypto
Successfully installed pycrypto

With tox and the patch rebased:

$ git init .
$ git fetch https://gerrit.wikimedia.org/r/operations/software/keyholder
$ git fetch https://gerrit.wikimedia.org/r/operations/software/keyholder refs/changes/36/458236/2 && git checkout FETCH_HEAD
$ GIT_COMMITTER_EMAIL=hashar@free.fr git rebase master
$ tox -e py34 --notest
...
Collecting pycrypto>=2.6 (from keyholder==0.1.dev44+gb004755)
  Running setup.py bdist_wheel for pycrypto ... done
  Stored in directory: /cache/pip/wheels/27/02/5e/77a69d0c16bb63c6ed32f5386f33a2809c94bd5414a2f6c196
...

I could not reproduce the issue so I went with live debugging in attempt to get the autoconf log file which is somewhere under /tmp.

To do so I created a change to point pip build dir to /log/tmp. Then crafted another change (461620) which depends on the previous one and based on the change that introduces pycrypto. Magically it passes.

I suspect that is because docker-registry.wikimedia.org/releng/tox:0.1.1 is run with --tmpfs /tmp and the tmpfs is mounted with the noexec flag.

The --tmpfs /tmp comes from abandoned change https://gerrit.wikimedia.org/r/#/c/integration/config/+/457070/ we did for T203181

hashar mentioned this in T203181: Quibble MariaDB should use a tmpfs as a datadir.Sep 20 2018, 11:20 AM

Mentioned in SAL (#wikimedia-releng) [2018-09-20T11:21:21Z] <hashar> Refreshing jenkins jobs to get rid of docker run option "--tmp /tmpfs" . It is mounted with 'noexec' which causes various jobs to fail. | T203181 and T204919

/tmp was mounted as a tmpfs which comes with the noexec flag. I guess autoconf compiles a binary and then try to execute it to validate the C compiler works, but the noexec flag prevent the execution and the test fails.

That is a leftover from an abandoned change for T203183, a lot of jobs (including tox-docker) did not get reverted properly. I have refreshed the job and it is no more using a tmpfs. A recheck on https://gerrit.wikimedia.org/r/#/c/operations/software/keyholder/+/458236/ pass with success.

@thcipriani from our discussion this week, the releng/tox container has libssl-dev, so we do not need an intermediate container that ship that dependency.

tox-docker fails on keyholder due to pycryptoClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

tox-docker fails on keyholder due to pycrypto
Closed, ResolvedPublic
Actions