We have to retire our Buster hosts , but search-loader's primary application (mjolnir) is currently configured to use Python 3.7 (the version that ships with Buster).
Bullseye ships with Python 3.9, so that's the minimum version we should be targeting. Creating this ticket to:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Source mjolnir deploy repo from gitlab | operations/puppet | production | +2 -1 |
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| search: Update mjolnir to python 3.10 | repos/data-engineering/airflow-dags!527 | ebernhardson | work/ebernhardson/mjolnir-py310 | main | |
| Update python to 3.10 | repos/search-platform/mjolnir!7 | ebernhardson | work/ebernhardson/libmamba | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Resolved | Gehel | T323921 [Epic] Migrate all Search Platform servers to Debian Bullseye | |||
| Resolved | bking | T346039 Migrate search-loader hosts to Bullseye or later | |||
| Open | None | T346189 Migrate search-loader into Kubernetes | |||
| Resolved | EBernhardson | T346373 Ensure mjolnir can work on Python 3.9 or later |
This needs an updated version of conda, attempts to update the python version currently result in the dependency resolver getting stuck. Will be resolved in T337258
The new version of conda-analytics-base is deployed everywhere now, so you should hopefully be unblocked from this. I'm leaving T337258 open whilst I troubleshoot an issue with stat1009, but the libmamba solver is ready to go by default.
btullis@stat1008:~$ source /usr/bin/conda-analytics-activate base
(base) btullis@stat1008:~$ conda info
active environment : base
active env location : /opt/conda-analytics
shell level : 1
user config file : /home/btullis/.condarc
populated config files : /opt/conda-analytics/condarc
/srv/home/btullis/.conda/condarc
conda version : 23.7.4
conda-build version : not installed
python version : 3.10.12.final.0
virtual packages : __archspec=1=x86_64
__glibc=2.28=0
__linux=5.10.0=0
__unix=0=0
base environment : /opt/conda-analytics (read only)
conda av data dir : /opt/conda-analytics/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/btullis/.conda/pkgs
/usr/lib/anaconda-wmf/pkgs
/opt/conda-analytics/pkgs
envs directories : /home/btullis/.conda/envs
/opt/conda-analytics/envs
platform : linux-64
user-agent : conda/23.7.4 requests/2.31.0 CPython/3.10.12 Linux/5.10.0-0.deb10.23-amd64 debian/10 glibc/2.28 solver/libmamba conda-libmamba-solver/23.7.0 libmambapy/1.5.1
UID:GID : 32741:500
netrc file : None
offline mode : False
(base) btullis@stat1008:~$ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/7
Update python to 3.10
Patch looks to work, but we will also want to check the model outputs after deploying the update.
ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/7
Update python to 3.10
ebernhardson opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/527
search: Update mjolnir to python 3.10
ebernhardson merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/527
search: Update mjolnir to python 3.10
Mentioned in SAL (#wikimedia-operations) [2023-10-24T19:56:36Z] <ebernhardson@deploy2002> Started deploy [airflow-dags/search@c585842]: T346373: Update mjolnir to use python 3.10
Mentioned in SAL (#wikimedia-operations) [2023-10-24T19:57:04Z] <ebernhardson@deploy2002> Finished deploy [airflow-dags/search@c585842]: T346373: Update mjolnir to use python 3.10 (duration: 00m 28s)
Change 973849 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):
[operations/puppet@production] Source mjolnir deploy repo from gitlab
ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/mjolnir-deploy/-/merge_requests/1
Update mjolnir to v2.4.0
ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/mjolnir-deploy/-/merge_requests/1
Update mjolnir to v2.4.0
The repo and the airflow side were updated, but we missed updating the search-loader daemons with the new version. Patches above address that issue.
Change 973849 merged by Bking:
[operations/puppet@production] Source mjolnir deploy repo from gitlab