Page MenuHomePhabricator
Create Task
Maniphest T337471

Jupyterhub is broken in conda-analytics 0.0.13
Open, HighPublic
Actions

Description

Whilst deploying a change to jupyterhub T336951: Reduce alert noise associated with individual users' jupyterhub-singleuser services the jupyterhub-conda services were restarted on all stat servers.

The service failed to restart, producing log entries like this:

May 25 11:24:42 stat1007 jupyterhub-conda[7958]: [I 2023-05-25 11:24:42.109 JupyterHub app:2479] Running JupyterHub version 1.5.0
May 25 11:24:42 stat1007 jupyterhub-conda[7958]: [I 2023-05-25 11:24:42.109 JupyterHub app:2509] Using Authenticator: builtins.PosixGroupCheckingAuthenticator
May 25 11:24:42 stat1007 jupyterhub-conda[7958]: [I 2023-05-25 11:24:42.109 JupyterHub app:2509] Using Spawner: spawners.CondaEnvProfilesSpawner
May 25 11:24:42 stat1007 jupyterhub-conda[7958]: [I 2023-05-25 11:24:42.109 JupyterHub app:2509] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.5.0
May 25 11:24:42 stat1007 jupyterhub-conda[7958]: [I 2023-05-25 11:24:42.113 JupyterHub app:1554] Loading cookie_secret from /srv/jupyterhub-conda/data/jupyterhub_cookie_secret
May 25 11:24:42 stat1007 jupyterhub-conda[7958]: [D 2023-05-25 11:24:42.113 JupyterHub app:1721] Connecting to db: sqlite:////srv/jupyterhub-conda/data/jupyterhub.sqlite.db
May 25 11:24:42 stat1007 jupyterhub-conda[7958]: [E 2023-05-25 11:24:42.123 JupyterHub app:2989]
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:     Traceback (most recent call last):
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/jupyterhub/app.py", line 2986, in launch_instance_async
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         await self.initialize(argv)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/jupyterhub/app.py", line 2521, in initialize
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         self.init_db()
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/jupyterhub/app.py", line 1726, in init_db
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         self.session_factory = orm.new_session_factory(
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/jupyterhub/orm.py", line 880, in new_session_factory
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         check_db_revision(engine)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/jupyterhub/orm.py", line 771, in check_db_revision
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         current_table_names = set(inspect(engine).get_table_names())
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/inspection.py", line 111, in inspect
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         ret = reg(subject)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 304, in _engine_insp
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         return Inspector._construct(Inspector._init_engine, bind)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 237, in _construct
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         init(self, bind)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 248, in _init_engine
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         engine.connect().close()
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3264, in connect
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         return self._connection_cls(self)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 174, in __init__
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         self.dispatch.engine_connect(self)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 487, in __call__
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         fn(*args, **kw)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/event/legacy.py", line 100, in wrap_leg
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         return fn(*conv(*args))
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/jupyterhub/orm.py", line 737, in ping_connection
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         connection.scalar(select([1]))
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/sql/_selectable_constructors.py", line 489, in select
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         return Select(*entities)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/sql/selectable.py", line 5132, in __init__
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         self._raw_columns = [
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/sql/selectable.py", line 5133, in <listcomp>
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         coercions.expect(
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/sql/coercions.py", line 413, in expect
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         resolved = impl._literal_coercion(
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/sql/coercions.py", line 652, in _literal_coercion
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         self._raise_for_expected(element, argname)
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/sql/coercions.py", line 1143, in _raise_for_expected
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         return super()._raise_for_expected(
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/sql/coercions.py", line 711, in _raise_for_expected
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         super()._raise_for_expected(
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:       File "/opt/conda-analytics/lib/python3.10/site-packages/sqlalchemy/sql/coercions.py", line 536, in _raise_for_expected
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:         raise exc.ArgumentError(msg, code=code) from err
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:     sqlalchemy.exc.ArgumentError: Column expression, FROM clause, or other columns clause element expected, got [1]. Did you mean to say select(1)?
May 25 11:24:42 stat1007 jupyterhub-conda[7958]:     
May 25 11:24:42 stat1007 jupyterhub-conda[7958]: [D 2023-05-25 11:24:42.127 JupyterHub application:1028] Exiting application: jupyterhub

The change to the jupyterhub configuration was reverted, but the services still didn't restart cleanly, suggesting that the change in T336951 may not have been the direct cause of the error.

We need to work on resolving this issue with a high priority.

Details

ReferenceSource BranchDest BranchAuthorTitle
repos/data-engineering/conda-analytics!31fix_conda_lockmainbtullisUse the conda environment lock file to generate the environment
Customize query in GitLab

Related Objects

Event Timeline

BTullis created this task.Thu, May 25, 11:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, May 25, 11:32 AM
BTullis triaged this task as Unbreak Now! priority.Thu, May 25, 11:32 AM
AndrewTavis_WMDE added a subscriber: AndrewTavis_WMDE.Thu, May 25, 11:40 AM
BTullis added a comment.Thu, May 25, 11:40 AM

The sqlite database file that is used by jupyterhub is present on the servers and hasn't been midified since the jupyterhub service was first restarted.

btullis@stat1007:~$ ls -l /srv/jupyterhub-conda/data/jupyterhub.sqlite.db
-rw-r--r-- 1 root root 204800 May 25 11:08 /srv/jupyterhub-conda/data/jupyterhub.sqlite.db

The python environment is conda-analytics.
The version on the stats hosts is 0.0.13, which was built two weeks ago.

btullis@stat1007:~$ apt-cache policy conda-analytics
conda-analytics:
  Installed: 0.0.13
  Candidate: 0.0.17
  Version table:
     0.0.17 1001
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages
 *** 0.0.13 100
        100 /var/lib/dpkg/status

I'm independently working on a new conda-analytics environment, so version 0.0.17 is ready to install, but hasn't been pushed out yet.

I wonder if there is an issue with version 0.0.13 somehow, which didn't get picked up at the time of the deploy.

BTullis added a comment.Thu, May 25, 11:47 AM

The most relevant change that I can see is this one: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/commit/25ea928a04903331953d1f25914984b94147f0d5#8ffde13df31f212345642ccfa4755f3bd99e7e61_90_107

image.png (139×677 px, 26 KB)

I'm going to try a test downgrade of conda-analytics on an-test-client1001 to see if this fixes the issue.

BTullis added a comment.Thu, May 25, 11:50 AM
btullis@an-test-client1001:~$ curl -o conda-analytics-0.0.12_amd64.deb https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/package_files/874/download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  922M  100  922M    0     0   104M      0  0:00:08  0:00:08 --:--:--  107M
btullis@an-test-client1001:~$ sudo dpkg -i conda-analytics-0.0.12_amd64.deb 
dpkg: warning: downgrading conda-analytics from 0.0.17 to 0.0.12
(Reading database ... 644774 files and directories currently installed.)
Preparing to unpack conda-analytics-0.0.12_amd64.deb ...
Unpacking conda-analytics (0.0.12) over (0.0.17) ...
BTullis added a subscriber: xcollazo.Thu, May 25, 11:57 AM

It worked, so I'm going to downgrade conda-analytics to version 0.0.12 on all stats servers. CC @xcollazo

Running the following to download the artifact to all stats servers.

btullis@cumin1001:~$ sudo cumin A:stat 'curl -o /tmp/conda-analytics-0.0.12_amd64.deb https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/package_files/874/download'
BTullis added a comment.Thu, May 25, 12:04 PM

Running the following to downgrade conda-analytics:

btullis@cumin1001:~$ sudo cumin A:stat 'dpkg -i /tmp/conda-analytics-0.0.12_amd64.deb'
BTullis added a comment.Thu, May 25, 12:09 PM

A temporary downgrading of the conda-analytics environment to version 0.0.12 will have a knock-on effect to the people working on Iceberg tickets, such as T335305: Migrate referrer_daily to Iceberg and T333013: Apache Iceberg Migration

It's possible that this change to the conda environment could also be related to T336800: platform_eng Airflow instance Spark jobs failing after Iceberg changes but I haven't looked at it in depth.

Stevemunene mentioned this in T336036: Bring stat1009 into service.Thu, May 25, 12:10 PM
JArguello-WMF edited projects, added Shared-Data-Infrastructure (Q4 Wrap up); removed Shared-Data-Infrastructure.Thu, May 25, 12:38 PM
JArguello-WMF moved this task from Next Up to Unexpected work/incident on the Shared-Data-Infrastructure (Q4 Wrap up) board.
BTullis lowered the priority of this task from Unbreak Now! to High.Thu, May 25, 1:10 PM

Lowering from unbreak now to high priority. We still need to work out how to get the iceberg jar into the conda-analytics environment, but without breaking jupyterhub. Maybe this will be a version 0.0.18 of conda-analytics.

BTullis renamed this task from Jupyterhub is broken with sqlalchemy error to Jupyterhub is broken in conda-analytics 0.0.13.Thu, May 25, 1:11 PM
BTullis added subscribers: mforns, JAllemandou.
BTullis added a comment.EditedThu, May 25, 1:24 PM

Here is where we pushed out conda-analytics 0.0.13 T335721: Add support for Iceberg in Spark

xcollazo added a comment.Thu, May 25, 1:51 PM

For the time being, for pyspark jobs in stat servers you can create sessions like this:

spark = wmfdata.spark.create_session(
    type='yarn-large',
    extra_settings={'spark.jars.packages' : 'org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2.1'}
)

The extra_settings will take care of downloading an extra jar related to our work on iceberg.

xcollazo added a comment.Thu, May 25, 2:10 PM
In T337471#8879708, @BTullis wrote:

The most relevant change that I can see is this one: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/commit/25ea928a04903331953d1f25914984b94147f0d5#8ffde13df31f212345642ccfa4755f3bd99e7e61_90_107

image.png (139×677 px, 26 KB)

I'm going to try a test downgrade of conda-analytics on an-test-client1001 to see if this fixes the issue.

I found the buster container version that was used to build conda-analytics 0.0.12:

REPOSITORY                                                          TAG               IMAGE ID       CREATED       SIZE
docker-registry.wikimedia.org/wikimedia-buster                      latest            29d44c40dcf7   3 days ago    69.3MB
conda-analytics                                                     latest            9ad224012ced   9 days ago    4.95GB
registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-6d07dd15   66e94e4eaa8d   3 weeks ago   67.1MB
docker-registry.wikimedia.org/buster                                latest            1adaf5125f5f   3 weeks ago   69.3MB      <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
docker-registry.wikimedia.org/wikimedia-buster                      <none>            1adaf5125f5f   3 weeks ago   69.3MB
registry.gitlab.com/gitlab-org/release-cli                          latest            9fef26e1927a   7 weeks ago   13MB

Source: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/jobs/30010#L515

Although, I think we should also look into the sqlachemy version as well since we know that the latest versions of that library has had compatibility issues, at least with Airflow.

AndrewTavis_WMDE removed a subscriber: AndrewTavis_WMDE.Thu, May 25, 2:10 PM
BTullis moved this task from Backlog to In Progress on the Data-Platform-SRE board.Thu, May 25, 7:28 PM
BTullis added a comment.EditedThu, May 25, 7:45 PM

I was wrong in my guess about the cause of the differences, it seems that it's not related to the update of buster.
I did a conda list in both the 0.0.12 and 0.0.13 versions of conda-analytics and there are big differences. Notably, sqlalchemy is upgraded from 1.4.44 to 2.0.15.

Edit: I made a mistake here This was the diff against versin 0.0.17, not 0.0.13. Still, I don't think it makes a material differnce. I will check the diff between 0.0.12 and 0.0.12 again.

Here's the full patch.

 --- /home/btullis/tmp/conda-analytics-0.0.12-conda.txt
+++ /home/btullis/tmp/conda-analytics-0.0.17-conda.txt
@@ -4,73 +4,75 @@
 _libgcc_mutex             0.1                 conda_forge    conda-forge
 _openmp_mutex             4.5                       2_gnu    conda-forge
 abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
-alembic                   1.8.1              pyhd8ed1ab_0    conda-forge
+alembic                   1.11.1             pyhd8ed1ab_0    conda-forge
 anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
 argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
 argon2-cffi-bindings      21.2.0          py310h5764c6d_3    conda-forge
 arrow-cpp                 8.0.0           py310hbe01597_5_cpu    conda-forge
-asttokens                 2.1.0              pyhd8ed1ab_0    conda-forge
+asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
 async_generator           1.10                       py_0    conda-forge
-attrs                     22.1.0             pyh71513ae_1    conda-forge
+attrs                     23.1.0             pyh71513ae_1    conda-forge
 aws-c-cal                 0.5.11               h95a6274_0    conda-forge
 aws-c-common              0.6.2                h7f98852_0    conda-forge
 aws-c-event-stream        0.2.7               h3541f99_13    conda-forge
 aws-c-io                  0.10.5               hfb6a706_0    conda-forge
 aws-checksums             0.1.11               ha31a3da_7    conda-forge
 aws-sdk-cpp               1.8.186              hecaee15_4    conda-forge
-babel                     2.11.0             pyhd8ed1ab_0    conda-forge
+babel                     2.12.1             pyhd8ed1ab_1    conda-forge
 backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
 backports                 1.0                pyhd8ed1ab_3    conda-forge
 backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
-beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
-bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
-blinker                   1.5                pyhd8ed1ab_0    conda-forge
-brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
+beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
+bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
+blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
+brotli                    1.0.9                h166bdaf_8    conda-forge
+brotli-bin                1.0.9                h166bdaf_8    conda-forge
 bzip2                     1.0.8                h7f98852_4    conda-forge
-c-ares                    1.18.1               h7f98852_0    conda-forge
-ca-certificates           2022.9.24            ha878542_0    conda-forge
-certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
+c-ares                    1.19.1               hd590300_0    conda-forge
+ca-certificates           2023.5.7             hbcca054_0    conda-forge
+certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
 certipy                   0.1.3                      py_0    conda-forge
-cffi                      1.15.1          py310h255011f_2    conda-forge
-charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
+cffi                      1.15.1          py310h255011f_3    conda-forge
+charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
 click                     8.1.3                    pypi_0    pypi
-colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
-comm                      0.1.0              pyhd8ed1ab_0    conda-forge
+comm                      0.1.3              pyhd8ed1ab_0    conda-forge
 conda                     4.13.0          py310hff52083_2    conda-forge
 conda-pack                0.7.0              pyh6c4a22f_0    conda-forge
-conda-package-handling    1.9.0           py310h5764c6d_1    conda-forge
-configurable-http-proxy   4.5.3                he2f69ee_3    conda-forge
-contourpy                 1.0.6                    pypi_0    pypi
-cryptography              38.0.3          py310h597c629_0    conda-forge
+conda-package-handling    2.0.2              pyh38be061_0    conda-forge
+conda-package-streaming   0.7.0              pyhd8ed1ab_1    conda-forge
+configurable-http-proxy   4.5.4                he2f69ee_2    conda-forge
+contourpy                 1.0.7                    pypi_0    pypi
+cryptography              39.0.0          py310h65dfdc0_0    conda-forge
 cycler                    0.11.0                   pypi_0    pypi
-debugpy                   1.6.3           py310hd8f1fbe_1    conda-forge
+debugpy                   1.6.7           py310heca2aa9_0    conda-forge
 decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
 defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
 entrypoints               0.4                pyhd8ed1ab_0    conda-forge
 executing                 1.2.0              pyhd8ed1ab_0    conda-forge
 findspark                 2.0.1                    pypi_0    pypi
-flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
-fonttools                 4.38.0                   pypi_0    pypi
-future                    0.18.2                   pypi_0    pypi
+flit-core                 3.9.0              pyhd8ed1ab_0    conda-forge
+fonttools                 4.39.4                   pypi_0    pypi
+future                    0.18.3                   pypi_0    pypi
 gflags                    2.2.2             he1b5a44_1004    conda-forge
 glog                      0.6.0                h6f12383_0    conda-forge
-greenlet                  2.0.1           py310hd8f1fbe_0    conda-forge
+greenlet                  2.0.2           py310hc6cd4ac_1    conda-forge
 grpc-cpp                  1.45.2               h9d3bbbb_5    conda-forge
 gssapi                    1.8.2                    pypi_0    pypi
 icu                       70.1                 h27087fc_0    conda-forge
 idna                      3.4                pyhd8ed1ab_0    conda-forge
-importlib-metadata        5.0.0              pyha770c72_1    conda-forge
-importlib_resources       5.10.0             pyhd8ed1ab_0    conda-forge
-ipykernel                 6.18.0             pyh210e3f2_0    conda-forge
-ipython                   8.6.0              pyh41d4057_1    conda-forge
+importlib-metadata        6.6.0              pyha770c72_0    conda-forge
+importlib_metadata        6.6.0                hd8ed1ab_0    conda-forge
+importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
+ipykernel                 6.23.1             pyh210e3f2_0    conda-forge
+ipython                   8.13.2             pyh41d4057_0    conda-forge
 ipython_genutils          0.2.0                      py_1    conda-forge
 jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
 jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
 json5                     0.9.5              pyh9f0ad1d_0    conda-forge
-jsonschema                4.17.1             pyhd8ed1ab_0    conda-forge
-jupyter_client            7.4.7              pyhd8ed1ab_0    conda-forge
-jupyter_core              5.0.0           py310hff52083_0    conda-forge
-jupyter_server            1.23.3             pyhd8ed1ab_0    conda-forge
+jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
+jupyter_client            8.2.0              pyhd8ed1ab_0    conda-forge
+jupyter_core              5.3.0           py310hff52083_0    conda-forge
+jupyter_server            1.23.6             pyhd8ed1ab_0    conda-forge
 jupyter_telemetry         0.1.0              pyhd8ed1ab_1    conda-forge
 jupyterhub                1.5.0           py310hff52083_1    conda-forge
 jupyterhub-base           1.5.0           py310hff52083_1    conda-forge
@@ -79,11 +81,11 @@
 jupyterhub-systemdspawner 0.15.0          py310hff52083_3    conda-forge
 jupyterlab                3.4.8              pyhd8ed1ab_0    conda-forge
 jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
-jupyterlab_server         2.16.3             pyhd8ed1ab_0    conda-forge
+jupyterlab_server         2.22.1             pyhd8ed1ab_0    conda-forge
 keyutils                  1.6.1                h166bdaf_0    conda-forge
 kiwisolver                1.4.4                    pypi_0    pypi
-krb5                      0.4.1                    pypi_0    pypi
-ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
+krb5                      0.5.0                    pypi_0    pypi
+ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
 ldap3                     2.9.1              pyhd8ed1ab_0    conda-forge
 libblas                   3.9.0           16_linux64_openblas    conda-forge
 libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
@@ -91,7 +93,7 @@
 libbrotlienc              1.0.9                h166bdaf_8    conda-forge
 libcblas                  3.9.0           16_linux64_openblas    conda-forge
 libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
-libcurl                   7.86.0               h7bff187_1    conda-forge
+libcurl                   7.87.0               h6312ad2_0    conda-forge
 libedit                   3.1.20191231         he28a2e2_2    conda-forge
 libev                     4.33                 h516909a_1    conda-forge
 libevent                  2.1.10               h9b69904_4    conda-forge
@@ -102,58 +104,56 @@
 libgomp                   12.2.0              h65d4601_19    conda-forge
 libgoogle-cloud           1.40.2               habd0e3a_0    conda-forge
 liblapack                 3.9.0           16_linux64_openblas    conda-forge
-libnghttp2                1.47.0               hdcd2b5c_1    conda-forge
+libnghttp2                1.51.0               hdcd2b5c_0    conda-forge
 libnsl                    2.0.0                h7f98852_0    conda-forge
 libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
-libprotobuf               3.20.2               h6239696_0    conda-forge
+libprotobuf               3.20.3               h3eb15da_0    conda-forge
 libsodium                 1.0.18               h36c2ea0_1    conda-forge
-libsqlite                 3.40.0               h753d276_0    conda-forge
+libsqlite                 3.42.0               h2797004_0    conda-forge
 libssh2                   1.10.0               haa6b8db_3    conda-forge
 libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
 libthrift                 0.16.0               h491838f_2    conda-forge
 libutf8proc               2.8.0                h166bdaf_0    conda-forge
-libuuid                   2.32.1            h7f98852_1000    conda-forge
+libuuid                   2.38.1               h0b41bf4_0    conda-forge
 libuv                     1.44.2               h166bdaf_0    conda-forge
 libzlib                   1.2.13               h166bdaf_4    conda-forge
-lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
+lz4-c                     1.9.4                hcb278e6_0    conda-forge
 mako                      1.2.4              pyhd8ed1ab_0    conda-forge
 mariadb                   1.0.11                   pypi_0    pypi
-markupsafe                2.1.1           py310h5764c6d_2    conda-forge
-matplotlib                3.6.2                    pypi_0    pypi
+markupsafe                2.1.2           py310h1fa729e_0    conda-forge
+matplotlib                3.7.1                    pypi_0    pypi
 matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
-mistune                   2.0.4              pyhd8ed1ab_0    conda-forge
-nbclassic                 0.4.8              pyhd8ed1ab_0    conda-forge
-nbclient                  0.7.0              pyhd8ed1ab_0    conda-forge
-nbconvert                 7.2.5              pyhd8ed1ab_0    conda-forge
-nbconvert-core            7.2.5              pyhd8ed1ab_0    conda-forge
-nbconvert-pandoc          7.2.5              pyhd8ed1ab_0    conda-forge
-nbformat                  5.7.0              pyhd8ed1ab_0    conda-forge
+mistune                   2.0.5              pyhd8ed1ab_0    conda-forge
+nbclassic                 1.0.0              pyh8b2e9e2_0    conda-forge
+nbclient                  0.8.0              pyhd8ed1ab_0    conda-forge
+nbconvert-core            7.4.0              pyhd8ed1ab_0    conda-forge
+nbformat                  5.8.0              pyhd8ed1ab_0    conda-forge
 ncurses                   6.3                  h27087fc_1    conda-forge
 nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
 nodejs                    18.12.1              h96d913c_0    conda-forge
-notebook                  6.5.2              pyha770c72_1    conda-forge
-notebook-shim             0.2.2              pyhd8ed1ab_0    conda-forge
+notebook                  6.5.4              pyha770c72_0    conda-forge
+notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
 numpy                     1.23.1          py310h53a5b5f_0    conda-forge
 oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
-openssl                   1.1.1s               h166bdaf_0    conda-forge
+openssl                   1.1.1t               h0b41bf4_0    conda-forge
 orc                       1.7.5                h6c59b99_0    conda-forge
-packaging                 21.3               pyhd8ed1ab_0    conda-forge
+packaging                 23.1               pyhd8ed1ab_0    conda-forge
 pamela                    1.0.0                      py_0    conda-forge
 pandas                    1.4.3           py310h769672d_0    conda-forge
-pandoc                    2.19.2               h32600fe_1    conda-forge
 pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
 parquet-cpp               1.5.1                         2    conda-forge
 parso                     0.8.3              pyhd8ed1ab_0    conda-forge
 pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
 pickleshare               0.7.5                   py_1003    conda-forge
-pillow                    9.3.0                    pypi_0    pypi
-pip                       22.3.1             pyhd8ed1ab_0    conda-forge
+pillow                    9.5.0                    pypi_0    pypi
+pip                       23.1.2             pyhd8ed1ab_0    conda-forge
 pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
-platformdirs              2.5.2              pyhd8ed1ab_1    conda-forge
+platformdirs              3.5.1              pyhd8ed1ab_0    conda-forge
 presto-python-client      0.8.3                    pypi_0    pypi
-prometheus_client         0.15.0             pyhd8ed1ab_0    conda-forge
-prompt-toolkit            3.0.33             pyha770c72_0    conda-forge
-psutil                    5.9.4           py310h5764c6d_0    conda-forge
+prometheus_client         0.16.0             pyhd8ed1ab_0    conda-forge
+prompt-toolkit            3.0.38             pyha770c72_0    conda-forge
+prompt_toolkit            3.0.38               hd8ed1ab_0    conda-forge
+psutil                    5.9.5           py310h1fa729e_0    conda-forge
 ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
 pure-sasl                 0.6.2                    pypi_0    pypi
 pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
@@ -163,61 +163,62 @@
 pycosat                   0.6.4           py310h5764c6d_1    conda-forge
 pycparser                 2.21               pyhd8ed1ab_0    conda-forge
 pycurl                    7.45.1          py310h2aed498_3    conda-forge
-pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
+pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
 pyhive                    0.6.5                    pypi_0    pypi
-pyjwt                     2.6.0              pyhd8ed1ab_0    conda-forge
-pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
-pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
-pyrsistent                0.19.2          py310h5764c6d_0    conda-forge
+pyjwt                     2.7.0              pyhd8ed1ab_0    conda-forge
+pyopenssl                 23.1.1             pyhd8ed1ab_0    conda-forge
+pyparsing                 3.0.9                    pypi_0    pypi
+pyrsistent                0.19.3          py310h1fa729e_0    conda-forge
 pysocks                   1.7.1              pyha2e5f31_6    conda-forge
 pyspark                   3.1.2              pyh6c4a22f_0    conda-forge
-pyspnego                  0.6.3                    pypi_0    pypi
+pyspnego                  0.9.0                    pypi_0    pypi
 python                    3.10.8          h257c98d_0_cpython    conda-forge
 python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
-python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
-python-json-logger        2.0.1              pyh9f0ad1d_0    conda-forge
+python-fastjsonschema     2.17.1             pyhd8ed1ab_0    conda-forge
+python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
 python_abi                3.10                    3_cp310    conda-forge
-pytz                      2022.6             pyhd8ed1ab_0    conda-forge
-pyzmq                     24.0.1          py310h330234f_1    conda-forge
+pytz                      2023.3             pyhd8ed1ab_0    conda-forge
+pyzmq                     25.0.2          py310h059b190_0    conda-forge
 re2                       2022.06.01           h27087fc_1    conda-forge
-readline                  8.1.2                h0f457ee_0    conda-forge
-requests                  2.28.1             pyhd8ed1ab_1    conda-forge
+readline                  8.2                  h8228510_1    conda-forge
+requests                  2.31.0             pyhd8ed1ab_0    conda-forge
 requests-kerberos         0.14.0                   pypi_0    pypi
-ruamel.yaml               0.17.21         py310h5764c6d_2    conda-forge
-ruamel.yaml.clib          0.2.7           py310h5764c6d_0    conda-forge
+ruamel.yaml               0.17.26         py310h2372a71_0    conda-forge
+ruamel.yaml.clib          0.2.7           py310h1fa729e_1    conda-forge
 ruamel_yaml               0.15.80         py310h5764c6d_1008    conda-forge
 s2n                       1.0.10               h9b69904_0    conda-forge
 sasl                      0.3.1                    pypi_0    pypi
-send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
-setuptools                65.5.1             pyhd8ed1ab_0    conda-forge
+send2trash                1.8.2              pyh41d4057_0    conda-forge
+setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
 six                       1.16.0             pyh6c4a22f_0    conda-forge
-snappy                    1.1.9                hbd366e4_2    conda-forge
+snappy                    1.1.10               h9fff704_0    conda-forge
 sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
 soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
-sqlalchemy                1.4.44          py310h5764c6d_0    conda-forge
-stack_data                0.6.1              pyhd8ed1ab_0    conda-forge
-terminado                 0.17.0             pyh41d4057_0    conda-forge
+sqlalchemy                2.0.15          py310h2372a71_0    conda-forge
+stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
+terminado                 0.17.1             pyh41d4057_0    conda-forge
 thrift                    0.16.0                   pypi_0    pypi
 thrift-sasl               0.4.3                    pypi_0    pypi
 tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
 tk                        8.6.12               h27826a3_0    conda-forge
 tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
 toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
-tornado                   6.2             py310h5764c6d_1    conda-forge
-tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
-traitlets                 5.5.0              pyhd8ed1ab_0    conda-forge
-typing_extensions         4.4.0              pyha770c72_0    conda-forge
-tzdata                    2022f                h191b570_0    conda-forge
-urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
-wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
+tornado                   6.3.2           py310h2372a71_0    conda-forge
+traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
+typing-extensions         4.5.0                hd8ed1ab_0    conda-forge
+typing_extensions         4.5.0              pyha770c72_0    conda-forge
+tzdata                    2023c                h71feb2d_0    conda-forge
+urllib3                   2.0.2              pyhd8ed1ab_0    conda-forge
+wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
 webencodings              0.5.1                      py_1    conda-forge
-websocket-client          1.4.2              pyhd8ed1ab_0    conda-forge
-wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
+websocket-client          1.5.2              pyhd8ed1ab_0    conda-forge
+wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
 wmfdata                   2.0.0                    pypi_0    pypi
 wrapspawner               1.0.1                    pypi_0    pypi
 xz                        5.2.6                h166bdaf_0    conda-forge
 yaml                      0.2.5                h7f98852_2    conda-forge
 zeromq                    4.3.4                h9c3ff4c_1    conda-forge
-zipp                      3.10.0             pyhd8ed1ab_0    conda-forge
+zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
 zlib                      1.2.13               h166bdaf_4    conda-forge
-zstd                      1.5.2                h6239696_4    conda-forge
+zstandard                 0.19.0          py310hdeb6495_1    conda-forge
+zstd                      1.5.2                h3eb15da_6    conda-forge
CodeReviewBot added a comment.Thu, May 25, 9:08 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/merge_requests/31

Use the conda environment lock file to generate the environment

CodeReviewBot added a project: Patch-For-Review.Thu, May 25, 9:08 PM
CodeReviewBot added a comment.Thu, May 25, 9:19 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/merge_requests/31

Use the conda environment lock file to generate the environment

Maintenance_bot removed a project: Patch-For-Review.Thu, May 25, 9:30 PM
BTullis added a comment.Thu, May 25, 10:09 PM

I've released version 0.0.18 of conda-analytics. Next I'm installing it to an-test-client1002 and I'll do another diff of conda-list.

btullis@an-test-client1002:~$ wget https://gitlab.wikimedia.org/api/v4/projects/359/packages/generic/conda-analytics/0.0.18/conda-analytics-0.0.18_amd64.deb
--2023-05-25 22:03:18--  https://gitlab.wikimedia.org/api/v4/projects/359/packages/generic/conda-analytics/0.0.18/conda-analytics-0.0.18_amd64.deb
Resolving gitlab.wikimedia.org (gitlab.wikimedia.org)... 2620:0:861:2:208:80:154:145, 208.80.154.145
Connecting to gitlab.wikimedia.org (gitlab.wikimedia.org)|2620:0:861:2:208:80:154:145|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1012586764 (966M) [application/octet-stream]
Saving to: ‘conda-analytics-0.0.18_amd64.deb’

conda-analytics-0.0.18_amd64.deb                     100%[=====================================================================================================================>] 965.68M   103MB/s    in 9.2s    

2023-05-25 22:03:28 (105 MB/s) - ‘conda-analytics-0.0.18_amd64.deb’ saved [1012586764/1012586764]
btullis@an-test-client1002:~$ sudo dpkg -i conda-analytics-0.0.18_amd64.deb 
(Reading database ... 187088 files and directories currently installed.)
Preparing to unpack conda-analytics-0.0.18_amd64.deb ...
Unpacking conda-analytics (0.0.18) over (0.0.17) ...
Setting up conda-analytics (0.0.18) ...
Post install script.
  Running /opt/conda-analytics/bin/python /opt/conda-analytics/bin/conda-unpack...
BTullis added a comment.Thu, May 25, 10:17 PM

OK, this looks much better. Some different build numbers, but the only python package difference between version 0.0.12 and 0.0.18 is the ipykernel [https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/commit/1ea5e1cef63cf103c228f03c3bb94df2645d2426#adf29efb8468a47aadba20034aeb4b4c50f5cda2_60_59|that we knew about]].

--- /home/btullis/tmp/conda-analytics-0.0.12-conda.txt
+++ /home/btullis/tmp/conda-analytics-0.0.18-conda.txt
@@ -31,7 +31,7 @@
 ca-certificates           2022.9.24            ha878542_0    conda-forge
 certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
 certipy                   0.1.3                      py_0    conda-forge
-cffi                      1.15.1          py310h255011f_2    conda-forge
+cffi                      1.15.1          py310h255011f_3    conda-forge
 charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
 click                     8.1.3                    pypi_0    pypi
 colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
@@ -61,7 +61,7 @@
 idna                      3.4                pyhd8ed1ab_0    conda-forge
 importlib-metadata        5.0.0              pyha770c72_1    conda-forge
 importlib_resources       5.10.0             pyhd8ed1ab_0    conda-forge
-ipykernel                 6.18.0             pyh210e3f2_0    conda-forge
+ipykernel                 6.19.4             pyh210e3f2_0    conda-forge
 ipython                   8.6.0              pyh41d4057_1    conda-forge
 ipython_genutils          0.2.0                      py_1    conda-forge
 jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
@@ -91,7 +91,7 @@
 libbrotlienc              1.0.9                h166bdaf_8    conda-forge
 libcblas                  3.9.0           16_linux64_openblas    conda-forge
 libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
-libcurl                   7.86.0               h7bff187_1    conda-forge
+libcurl                   7.86.0               h6312ad2_2    conda-forge
 libedit                   3.1.20191231         he28a2e2_2    conda-forge
 libev                     4.33                 h516909a_1    conda-forge
 libevent                  2.1.10               h9b69904_4    conda-forge
@@ -107,7 +107,7 @@
 libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
 libprotobuf               3.20.2               h6239696_0    conda-forge
 libsodium                 1.0.18               h36c2ea0_1    conda-forge
-libsqlite                 3.40.0               h753d276_0    conda-forge
+libsqlite                 3.40.0               h753d276_1    conda-forge
 libssh2                   1.10.0               haa6b8db_3    conda-forge
 libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
 libthrift                 0.16.0               h491838f_2    conda-forge
@@ -135,12 +135,12 @@
 notebook-shim             0.2.2              pyhd8ed1ab_0    conda-forge
 numpy                     1.23.1          py310h53a5b5f_0    conda-forge
 oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
-openssl                   1.1.1s               h166bdaf_0    conda-forge
+openssl                   1.1.1s               h0b41bf4_1    conda-forge
 orc                       1.7.5                h6c59b99_0    conda-forge
 packaging                 21.3               pyhd8ed1ab_0    conda-forge
 pamela                    1.0.0                      py_0    conda-forge
 pandas                    1.4.3           py310h769672d_0    conda-forge
-pandoc                    2.19.2               h32600fe_1    conda-forge
+pandoc                    2.19.2               h32600fe_2    conda-forge
 pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
 parquet-cpp               1.5.1                         2    conda-forge
 parso                     0.8.3              pyhd8ed1ab_0    conda-forge
@@ -180,11 +180,11 @@
 pytz                      2022.6             pyhd8ed1ab_0    conda-forge
 pyzmq                     24.0.1          py310h330234f_1    conda-forge
 re2                       2022.06.01           h27087fc_1    conda-forge
-readline                  8.1.2                h0f457ee_0    conda-forge
+readline                  8.1.2                h8228510_1    conda-forge
 requests                  2.28.1             pyhd8ed1ab_1    conda-forge
 requests-kerberos         0.14.0                   pypi_0    pypi
-ruamel.yaml               0.17.21         py310h5764c6d_2    conda-forge
-ruamel.yaml.clib          0.2.7           py310h5764c6d_0    conda-forge
+ruamel.yaml               0.17.21         py310h1fa729e_3    conda-forge
+ruamel.yaml.clib          0.2.7           py310h1fa729e_1    conda-forge
 ruamel_yaml               0.15.80         py310h5764c6d_1008    conda-forge
 s2n                       1.0.10               h9b69904_0    conda-forge
 sasl                      0.3.1                    pypi_0    pypi
@@ -220,4 +220,4 @@
 zeromq                    4.3.4                h9c3ff4c_1    conda-forge
 zipp                      3.10.0             pyhd8ed1ab_0    conda-forge
 zlib                      1.2.13               h166bdaf_4    conda-forge
-zstd                      1.5.2                h6239696_4    conda-forge
+zstd                      1.5.2                h3eb15da_6    conda-forge

I'll push out this version to the stats servers now, so that it fixes the jupyterhub issue.

We can then decide whether it's worth pushing out this version to the rest of the servers tomorrow, or wait until next week.

BTullis added a comment.Thu, May 25, 10:25 PM

Pulled the package to the apt server:

btullis@apt1001:~$ wget https://gitlab.wikimedia.org/api/v4/projects/359/packages/generic/conda-analytics/0.0.18/conda-analytics-0.0.18_amd64.deb
--2023-05-25 22:17:41--  https://gitlab.wikimedia.org/api/v4/projects/359/packages/generic/conda-analytics/0.0.18/conda-analytics-0.0.18_amd64.deb
Resolving gitlab.wikimedia.org (gitlab.wikimedia.org)... 2620:0:861:2:208:80:154:145, 208.80.154.145
Connecting to gitlab.wikimedia.org (gitlab.wikimedia.org)|2620:0:861:2:208:80:154:145|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1012586764 (966M) [application/octet-stream]
Saving to: ‘conda-analytics-0.0.18_amd64.deb’

conda-analytics-0.0.18_amd64.deb                     100%[=====================================================================================================================>] 965.68M  97.0MB/s    in 10s     

2023-05-25 22:17:51 (96.2 MB/s) - ‘conda-analytics-0.0.18_amd64.deb’ saved [1012586764/1012586764]

btullis@apt1001:~$ dpkg-deb --info conda-analytics-0.0.18_amd64.deb 
 new Debian package, version 2.0.
 size 1012586764 bytes: control archive=940 bytes.
     570 bytes,    13 lines      control              
      49 bytes,     1 lines      files                
     645 bytes,    19 lines   *  postinst             #!/usr/bin/env
 Package: conda-analytics
 Version: 0.0.18
 Architecture: amd64
 Maintainer: Aqu (WMF) <aquhen@wikimedia.org>
 Installed-Size: 3716833
 Depends: bash, default-jre-headless | java8-runtime-headless, libsasl2-2, libmariadb-dev
 Section: python
 Priority: optional
 Homepage: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics
 Description: conda packed environment with pyspark for WMF
  This package contains the standalone environment conda-analytics.
  This package is intended to be installed across the analytics cluster to provide Spark 3 and other
  libraries.

Updated the version of conda-analytics on apt.wikimedia.org

btullis@apt1001:~$ sudo -i reprepro includedeb buster-wikimedia `pwd`/conda-analytics-0.0.18_amd64.deb
Exporting indices...
btullis@apt1001:~$ sudo -i reprepro includedeb bullseye-wikimedia `pwd`/conda-analytics-0.0.18_amd64.deb
Exporting indices...
Deleting files no longer referenced...

Installing it on the stats servers. I have already run apt update on them all to get the latest package lists.

btullis@cumin1001:~$ sudo cumin A:stat 'apt install conda-analytics'
6 hosts will be targeted:
stat[1004-1009].eqiad.wmnet
OK to proceed on 6 hosts? Enter the number of affected hosts to confirm or "q" to quit: 6
BTullis added a comment.Thu, May 25, 10:35 PM

I've restarted the jupyterhub-conda service on stat1008 and connected successfully, so I think this is now fixed.

xcollazo awarded a token.Fri, May 26, 3:12 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL