| mpopov | |
| Thu, Nov 21, 3:13 PM |
| F57729449: Screenshot 2024-11-21 at 10.11.54 AM.png | |
| Thu, Nov 21, 3:13 PM |
Steps to replicate: File > Hub Control Panel > Stop My Server > Start My Server > Create and use new cloned conda environment
Consistently getting the same error on stat1010 and stat1011
Also reported by @tchin about running into this issue yesterday but not earlier today.
Currently trying this out on stat1010, creating a new clonled environment and tailing the systemctl unit log file.
btullis@stat1010:~$ journalctl -u jupyter-btullis-singleuser-conda-analytics.service -f -- Journal begins at Thu 2024-11-21 05:09:27 UTC. -- Nov 21 16:52:35 stat1010 systemd[1]: Started /bin/bash -c cd /home/btullis && exec /etc/jupyterhub-conda/jupyterhub-singleuser-conda-env.sh __NEW__ --port=48013 --SingleUserNotebookApp.default_url=/lab. Nov 21 16:52:35 stat1010 jupyterhub-conda-singleuser[540711]: Creating new cloned conda env 2024-11-21T16.52.35_btullis... Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: Source: /opt/conda-analytics Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: Destination: /home/btullis/.conda/envs/2024-11-21T16.52.35_btullis Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: The following packages cannot be cloned out of the root environment: Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: - conda-forge/linux-64::conda-23.10.0-py310hff52083_1 Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: - conda-forge/noarch::conda-libmamba-solver-24.7.0-pyhd8ed1ab_0 Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: Packages: 225 Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: Files: 1255 Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: Downloading and Extracting Packages: ...working... done Nov 21 16:52:39 stat1010 jupyterhub-conda-singleuser[540732]: Downloading and Extracting Packages: ...working... done Nov 21 16:52:40 stat1010 jupyterhub-conda-singleuser[540732]: Preparing transaction: ...working... done Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540732]: # Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540732]: # To activate this environment, use Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540732]: # Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540732]: # $ conda activate 2024-11-21T16.52.35_btullis Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540732]: # Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540732]: # To deactivate an active environment, use Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540732]: # Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540732]: # $ conda deactivate Nov 21 16:52:59 stat1010 jupyterhub-conda-singleuser[540711]: Installing the conda and conda-libmamba-solver packages to the newly cloned environment: 2024-11-21T16.52.35_btullis Nov 21 16:53:00 stat1010 jupyterhub-conda-singleuser[540785]: Channels: Nov 21 16:53:00 stat1010 jupyterhub-conda-singleuser[540785]: - defaults Nov 21 16:53:00 stat1010 jupyterhub-conda-singleuser[540785]: Platform: linux-64 Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: Collecting package metadata (repodata.json): ...working... WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found> Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: done Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: Solving environment: ...working... done Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: ## Package Plan ## Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: environment location: /home/btullis/.conda/envs/2024-11-21T16.52.35_btullis Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: added / updated specs: Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: - conda-libmamba-solver=24.7.0 Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: - conda=23.10.0 Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: The following NEW packages will be INSTALLED: Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: conda conda-forge/linux-64::conda-23.10.0-py310hff52083_1 Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: conda-libmamba-so~ conda-forge/noarch::conda-libmamba-solver-24.7.0-pyhd8ed1ab_0 Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: Downloading and Extracting Packages: ...working... done Nov 21 16:53:02 stat1010 jupyterhub-conda-singleuser[540785]: Preparing transaction: ...working... done Nov 21 16:53:03 stat1010 jupyterhub-conda-singleuser[540785]: Verifying transaction: ...working... done Nov 21 16:53:03 stat1010 jupyterhub-conda-singleuser[540785]: Executing transaction: ...working... done Nov 21 16:53:04 stat1010 jupyterhub-conda-singleuser[540711]: Thu 21 Nov 2024 04:53:04 PM UTC Created user conda environment 2024-11-21T16.52.35_btullis Nov 21 16:53:04 stat1010 jupyterhub-conda-singleuser[540711]: To activate this environment with vanilla conda run: Nov 21 16:53:04 stat1010 jupyterhub-conda-singleuser[540711]: source /opt/conda-analytics/etc/profile.d/conda.sh Nov 21 16:53:04 stat1010 jupyterhub-conda-singleuser[540711]: conda activate 2024-11-21T16.52.35_btullis Nov 21 16:53:04 stat1010 jupyterhub-conda-singleuser[540711]: Alternatively, you can use the conda-analytic helper script: Nov 21 16:53:04 stat1010 jupyterhub-conda-singleuser[540711]: source conda-analytics-activate 2024-11-21T16.52.35_btullis
This succeeded for me.
Checking the output from journalctl -u jupyter-bearloga-singleuser-conda-analytics.service
Nov 21 14:53:25 stat1010 jupyterhub-conda-singleuser[493192]: Creating new cloned conda env 2024-11-21T14.53.25_bearloga...
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: Source: /opt/conda-analytics
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: Destination: /home/bearloga/.conda/envs/2024-11-21T14.53.25_bearloga
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: The following packages cannot be cloned out of the root environment:
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: - conda-forge/linux-64::conda-23.10.0-py310hff52083_1
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: - conda-forge/noarch::conda-libmamba-solver-24.7.0-pyhd8ed1ab_0
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: Packages: 225
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: Files: 1255
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: Downloading and Extracting Packages: ...working... done
Nov 21 14:53:31 stat1010 jupyterhub-conda-singleuser[493212]: Downloading and Extracting Packages: ...working... done
Nov 21 14:53:32 stat1010 jupyterhub-conda-singleuser[493212]: Preparing transaction: ...working... done
Nov 21 14:53:37 stat1010 jupyterhub-conda-singleuser[493212]: Verifying transaction: ...working... done
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: Executing transaction: ...working... done
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: #
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: # To activate this environment, use
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: #
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: # $ conda activate 2024-11-21T14.53.25_bearloga
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: #
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: # To deactivate an active environment, use
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: #
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493212]: # $ conda deactivate
Nov 21 14:53:48 stat1010 jupyterhub-conda-singleuser[493192]: Installing the conda and conda-libmamba-solver packages to the newly cloned environment: 2024-11-21T14.53.25_bearloga
Nov 21 14:53:49 stat1010 jupyterhub-conda-singleuser[493333]: Channels:
Nov 21 14:53:49 stat1010 jupyterhub-conda-singleuser[493333]: - defaults
Nov 21 14:53:49 stat1010 jupyterhub-conda-singleuser[493333]: Platform: linux-64
Nov 21 14:53:50 stat1010 jupyterhub-conda-singleuser[493333]: Collecting package metadata (repodata.json): ...working... done
Nov 21 14:53:51 stat1010 jupyterhub-conda-singleuser[493333]: Solving environment: ...working... done
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/noarch/conda-libmamba-solver-24.7.0-pyhd3eb1b0_0.conda\nThis >
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/linux-64/certifi-2024.8.30-py310h06a4308_0.conda\nThis comman>
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/linux-64/ca-certificates-2024.9.24-h06a4308_0.conda\nThis com>
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: ## Package Plan ##
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: environment location: /home/bearloga/.conda/envs/2024-11-21T14.53.25_bearloga
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: added / updated specs:
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: - conda-libmamba-solver=24.7.0
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: - conda=23.10.0
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: The following packages will be downloaded:
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: package | build
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: ---------------------------|-----------------
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: ca-certificates-2024.9.24 | h06a4308_0 130 KB defaults
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: certifi-2024.8.30 | py310h06a4308_0 162 KB defaults
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: conda-libmamba-solver-24.7.0| pyhd3eb1b0_0 39 KB defaults
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: ------------------------------------------------------------
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: Total: 332 KB
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: The following NEW packages will be INSTALLED:
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: conda conda-forge/linux-64::conda-23.10.0-py310hff52083_1
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: conda-libmamba-so~ pkgs/main/noarch::conda-libmamba-solver-24.7.0-pyhd3eb1b0_0
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: The following packages will be UPDATED:
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: ca-certificates conda-forge::ca-certificates-2024.8.3~ --> pkgs/main::ca-certificates-2024.9.24-h06a4308_0
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: The following packages will be SUPERSEDED by a higher-priority channel:
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: certifi conda-forge/noarch::certifi-2024.8.30~ --> pkgs/main/linux-64::certifi-2024.8.30-py310h06a4308_0
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: Downloading and Extracting Packages: ...working... doneSo the difference would seem to be that in @mpopov's environment we see:
Nov 21 14:53:50 stat1010 jupyterhub-conda-singleuser[493333]: Collecting package metadata (repodata.json): ...working... done
Nov 21 14:53:51 stat1010 jupyterhub-conda-singleuser[493333]: Solving environment: ...working... done
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/noarch/conda-libmamba-solver-24.7.0-pyhd3eb1b0_0.conda\nThis >
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/linux-64/certifi-2024.8.30-py310h06a4308_0.conda\nThis comman>
Nov 21 14:53:52 stat1010 jupyterhub-conda-singleuser[493333]: RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/linux-64/ca-certificates-2024.9.24-h06a4308_0.conda\nThis com>Whereas in mine we see:
Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: Collecting package metadata (repodata.json): ...working... WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found> Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. Nov 21 16:53:01 stat1010 jupyterhub-conda-singleuser[540785]: WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail.
I can reproduce this on the terminal.
btullis@stat1010:~$ sudo -u bearloga bash
bearloga@stat1010:/srv/home/btullis$ conda-analytics-clone btullis-T380477
<snip snip>
Installing the conda and conda-libmamba-solver packages to the newly cloned environment: btullis-T380477
Channels:
- defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/bearloga/.conda/envs/btullis-T380477
added / updated specs:
- conda-libmamba-solver=24.7.0
- conda=23.10.0
The following packages will be downloaded:
package | build
---------------------------|-----------------
ca-certificates-2024.9.24 | h06a4308_0 130 KB defaults
certifi-2024.8.30 | py310h06a4308_0 162 KB defaults
conda-libmamba-solver-24.7.0| pyhd3eb1b0_0 39 KB defaults
------------------------------------------------------------
Total: 332 KB
The following NEW packages will be INSTALLED:
conda conda-forge/linux-64::conda-23.10.0-py310hff52083_1
conda-libmamba-so~ pkgs/main/noarch::conda-libmamba-solver-24.7.0-pyhd3eb1b0_0
The following packages will be UPDATED:
ca-certificates conda-forge::ca-certificates-2024.8.3~ --> pkgs/main::ca-certificates-2024.9.24-h06a4308_0
The following packages will be SUPERSEDED by a higher-priority channel:
certifi conda-forge/noarch::certifi-2024.8.30~ --> pkgs/main/linux-64::certifi-2024.8.30-py310h06a4308_0
Downloading and Extracting Packages:
RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/noarch/conda-libmamba-solver-24.7.0-pyhd3eb1b0_0.conda\nThis command is using a remote connection in offline mode.\n')
RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/linux-64/ca-certificates-2024.9.24-h06a4308_0.conda\nThis command is using a remote connection in offline mode.\n')
RuntimeError('EnforceUnusedAdapter called with url https://repo.anaconda.com/pkgs/main/linux-64/certifi-2024.8.30-py310h06a4308_0.conda\nThis command is using a remote connection in offline mode.\n')
bearloga@stat1010:/srv/home/btullis$@mpopov - It seems to be something to do with the .conda/pkgs directory in your home directory.
I renamed the directory and tried again and it worked.
bearloga@stat1010:~$ mv .conda/pkgs/ .conda/pks-bak bearloga@stat1010:~$ conda-analytics-clone btullis-T380477 <snip snip> Collecting package metadata (repodata.json): / WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. WARNING conda.conda_libmamba_solver.index:_json_path_to_repo_info(281): No repodata found for channel defaults. Solve will fail. done Solving environment: done <snip snip> Fri 22 Nov 2024 06:07:22 PM UTC Created user conda environment btullis-T380477 To activate this environment with vanilla conda run: source /opt/conda-analytics/etc/profile.d/conda.sh conda activate btullis-T380477 Alternatively, you can use the conda-analytic helper script: source conda-analytics-activate btullis-T380477
Maybe it was only the file: .conda/pkgs/urls.txt which contained lots of references to https://repo.anaconda.com that was confusing it.
After I ran the command above, the new .conda/pkgs directory looks as follows:
bearloga@stat1010:~$ ls -l .conda/pkgs/ total 4 drwxrwsr-x 2 bearloga wikidev 4096 Nov 22 18:07 cache -rw-r--r-- 1 bearloga wikidev 0 Nov 22 18:07 urls -rw-r--r-- 1 bearloga wikidev 0 Nov 22 18:07 urls.txt
Are you OK with this solution of removing or backing up ~/.conda/pkgs on hosts where you are experiencing this issue?
I can ask the same question to @tchin - if you're still experiencing this issue, too.
I'm not sure what would cause the ~/.conda/pkgs/urls.txt to be populated again, so maybe this issue will happen again under some condition.
Or maybe if we remove that directory once, it will fix the issue.
Tentatively resolving the issue, but please do feel free to reopen it @mpopov if things are not improved by this.
Thank you so much, Ben!!! Can confirm that mv .conda/pkgs/ .conda/pkgs-bak on stat1008–1011 fixed it.
I just now ran into the same issue, and deleting .conda/pkgs fixed it for me as well (in fact, I suspect I've been running into this issue sporadically for weeks). Thank you for figuring it out, @BTullis!
I've added the workaround to the docs: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Jupyter#Spawn_failure_after_creating_a_new_environment.