Currently we have a virtualenv that also uses the system packages, and install things like numpy / scipy from debian and everything else from pypi.
Instead we should use a fully isolated virtualenv that only installs things from the wheels we build.