Page MenuHomePhabricator

Cannot import Numpy in new Conda environment on stat1008
Closed, DeclinedPublic

Description

In a newly-created Conda environment on stat1008, I cannot import Numpy. After Conda-installing Numpy in my stacked environment, I get a different error but it still doesn't work. This also means I cannot import Pandas, Matplotlib, or Wmfdata-Python since those all depend on Numpy.

In one previous attempt Conda-installing Numpy fixed it, so I was able to try importing Pandas and Matplotlib. Both of those then had their own import errors, but now I can't replicate that since they just give Numpy import error.

I've tried on stat1006 and stat1007, and things work correctly there so it must be something specific to that machine.

Errors and output

neilpquinn-wmf@stat1008:~$ conda list numpy
# packages in environment at /home/neilpquinn-wmf/.conda/envs/2022-07-18T17.28.15_neilpquinn-wmf:
#
# Name                    Version                   Build  Channel
neilpquinn-wmf@stat1008:~$ conda list -n base numpy
# packages in environment at /usr/lib/anaconda-wmf:
#
# Name                    Version                   Build  Channel
numpy                     1.18.1           py37h95a1406_0    conda-forge/label/cf202003

The import error I get:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
File /usr/lib/anaconda-wmf/lib/python3.7/site-packages/numpy/core/__init__.py:24, in <module>
     23 try:
---> 24     from . import multiarray
     25 except ImportError as exc:

File /usr/lib/anaconda-wmf/lib/python3.7/site-packages/numpy/core/multiarray.py:14, in <module>
     12 import sys
---> 14 from . import overrides
     15 from . import _multiarray_umath

File /usr/lib/anaconda-wmf/lib/python3.7/site-packages/numpy/core/overrides.py:7, in <module>
      5 import textwrap
----> 7 from numpy.core._multiarray_umath import (
      8     add_docstring, implement_array_function, _get_implementing_args)
      9 from numpy.compat._inspect import getargspec

ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 import numpy

File /usr/lib/anaconda-wmf/lib/python3.7/site-packages/numpy/__init__.py:142, in <module>
    139 # Allow distributors to run custom init code
    140 from . import _distributor_init
--> 142 from . import core
    143 from .core import *
    144 from . import compat

File /usr/lib/anaconda-wmf/lib/python3.7/site-packages/numpy/core/__init__.py:54, in <module>
     26     import sys
     27     msg = """
     28 
     29 IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
   (...)
     52 """ % (sys.version_info[0], sys.version_info[1], sys.executable,
     53         __version__, exc)
---> 54     raise ImportError(msg)
     55 finally:
     56     for envkey in env_added:

ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.9 from "/home/neilpquinn-wmf/.conda/envs/2022-07-18T17.28.15_neilpquinn-wmf/bin/python",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.18.1" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

Output from installing Numpy in my stacked environment:

neilpquinn-wmf@stat1008:~$ conda install numpy
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.11.0
  latest version: 4.13.0

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /home/neilpquinn-wmf/.conda/envs/2022-07-18T17.54.26_neilpquinn-wmf

  added / updated specs:
    - numpy


The following NEW packages will be INSTALLED:

  blas               pkgs/main/linux-64::blas-1.0-mkl
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561
  mkl                pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640
  mkl-service        pkgs/main/linux-64::mkl-service-2.4.0-py39h7f8727e_0
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.3.1-py39hd3c417c_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.2.2-py39h51133e4_0
  numpy              pkgs/main/linux-64::numpy-1.22.3-py39he7a7128_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.22.3-py39hf524024_0


Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Now there's a new error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 import numpy

File ~/.conda/envs/2022-07-18T17.54.26_neilpquinn-wmf/lib/python3.9/site-packages/numpy/__init__.py:112, in <module>
    109 import sys
    110 import warnings
--> 112 from ._globals import (
    113     ModuleDeprecationWarning, VisibleDeprecationWarning,
    114     _NoValue, _CopyMode
    115 )
    117 # We first need to detect if we're being called as part of the numpy setup
    118 # procedure itself in a reliable manner.
    119 try:

ImportError: cannot import name '_CopyMode' from 'numpy._globals' (/usr/lib/anaconda-wmf/lib/python3.7/site-packages/numpy/_globals.py)

Event Timeline

Can't reproduce:

[urbanecm@stat1008 ~]$ conda-created-stacked
Creating new stacked conda env 2022-07-18T22.46.23_urbanecm...
[...]
[urbanecm@stat1008 ~]$ source conda-activate-stacked
Activating /home/urbanecm/.conda/envs/2022-07-18T22.46.23_urbanecm stacked on /usr/lib/anaconda-wmf...
Mon 18 Jul 2022 10:47:27 PM UTC Activated conda environment in /home/urbanecm/.conda/envs/2022-07-18T22.46.23_urbanecm stacked on /usr/lib/anaconda-wmf.
[...]
(2022-07-18T22.46.23_urbanecm) [urbanecm@stat1008 ~]$ python3
Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.linspace(0, 49)
array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
       13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
       26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
       39., 40., 41., 42., 43., 44., 45., 46., 47., 48., 49.])
>>>
(2022-07-18T22.46.23_urbanecm) [urbanecm@stat1008 ~]$ source deactivate
Deactivating /home/urbanecm/.conda/envs/2022-07-18T22.46.23_urbanecm (stack depth: 2).
Deactivating /usr/lib/anaconda-wmf (stack depth: 1).
[urbanecm@stat1008 ~]$

FTR, I did not have to install numpy manually (numpy and few other useful packages are a part of the base environment, /usr/lib/anaconda-wmf). I also didn't have any environment installed from the past (I normally use a different statbox).

@Urbanecm thanks for checking that! Generally, I would have assumed that I had broken something but as far as I knew, creating a new (stacked) Conda environment should act as a factory reset.

But apparently that's not the case; I tried removing everything specific to my environment by just deleting everything in my home directly, including hidden files (obviously, I backed up everything first). Now, after creating a new environment, everything works properly.

Ultimately, I suspect this came from the stacking not working properly, so the ultimate solution is probably T302819.

I'm glad to hear it works for you now @nshahquinn-wmf. Perhaps, deleting ~/.conda might be enough (as that's where Anaconda's supposed to store its files), but looks we can't test that (until it breaks again :)).

Perhaps, deleting ~/.conda might be enough (as that's where Anaconda's supposed to store its files), but looks we can't test that (until it breaks again :)).

That's actually a good point; I should've tried that. I guess I was annoyed, so I jumped to the most drastic option immediately 😂

I updated my troubleshooting instructions to suggest the less extreme option first.