Also worth noting that version >= 6 are not currently working with spicerack (T328775)

Ran into this today trying to pip install wikimedia-spicerack (Python 3.11).

Worked around it with pip install "pyyaml<5" wikimedia-spicerack

@colewhite from your comment on on the elastic search restart cookbook can i assume that moving to the opensearch python packages would also allow us to drop this dependency from spicerack. i.e. we could use that package for the current search cookbooks and this new ones?

In T345337#9150415, @jbond wrote:

@colewhite from your comment on on the elastic search restart cookbook can i assume that moving to the opensearch python packages would also allow us to drop this dependency from spicerack. i.e. we could use that package for the current search cookbooks and this new ones?

Spicerack uses curator to change cluster routing settings (initial patch, code) which is an important step to doing rolling maintenance. To remove the dependency, I would either bring in the ClusterRouting class or reimplement the calls to change allocation settings using the OpenSearch client directly (client.cluster.put_settings()). Either option should be fairly simple to do.

I don't recommended using our fork of curator as we'd still have to contend with the dependencies.

opensearch-py should be compatible with our elasticsearch 7.x instances. It was forked from the 7.10 release of elasticsearch-py.

@colewhite thanks

Hmm, elasticsearch-curator also tries to build and install pyyaml 3 if it's not there, and failing for me on python3.11. Note that I pinned pyyaml to 5.3.1, to overcome a cython bug (https://github.com/yaml/pyyaml/issues/601), but the last stack trace, is elascitsearch-curator trying to install pyyaml 3.13 and failing due to being unable to build the wheel :/

Even though that header is there:

dcaro@urcuchillay$ ls -la /usr/include/python3.11/cpython/longintrepr.h
-rw-r--r--. 1 root root 3817 Oct  2 15:29 /usr/include/python3.11/cpython/longintrepr.h

Change 966492 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/software/spicerack@master] elasticsearch: move to opensearch client

https://gerrit.wikimedia.org/r/966492

Related task T345900: Add a dependency on the opensearch-py client

This unblocked this issue and made tox pass:
https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/967244

The move out of elasticsearch-curator is still nice, but does not block this anymore :)

Unfortunately, I think this specific bug still exists, because there's no Python 3.11 wheel in PyPI: https://pypi.org/project/PyYAML/5.4.1/#files

Tox is now working on the CI where we use Python 3.9, but if I try to run pip install wikimedia-spicerack on a clean Python 3.11 Docker image, it tries to build PyYAML 5.4.1 and it fails with the Cython error:

~ $ docker run --rm -it python:3.11 bash
root@84be1c63f10b:/# apt update
root@84be1c63f10b:/# apt install pip

...

Collecting pyyaml>=3.12 (from kubernetes==12.0.*->wikimedia-spicerack)
  Downloading PyYAML-5.4.1.tar.gz (175 kB)

...

      AttributeError: cython_sources

Running tox on the wmcs-cookbooks repo fails with the same error, if I run it from a clean python:3.11 Docker image.

Locally on my machine (without Docker), it works only because I have a cached wheel in my pip cache, that I built before Cython 3.0 was released.

yep, I tried a few combinations of elasticsearch-curator/pyyaml and such... it turns out that elasticseach-curator dependency on pyyaml is 3.13, until 5.8.4, where it jumps to pinning it to 5.4.1, and none of those have wheels for 3.11, it wasn't until pyyaml 6.x that they starting building those :/

Note that both elasticsearch-curator and pyyaml 5 are pretty old.

So yep, still blocked 😢

Currently the only workaround I've found (as we don't use elasticsearch itself) is to install in the local venv the spicerack with the patch (https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/966492/) applied

Note that as of jan 2024, you will need also to workaround that python-kafka<=2.0.2 does not work with python 3.12, 2.0.3 will have the fix but it's not released yet (https://github.com/dpkp/kafka-python/issues/2412), you can pip install git+https://github.com/dpkp/kafka-python.git (https://github.com/dpkp/kafka-python/pull/2398#issuecomment-1797123710)

@fnegri Thanks a lot for resuming this and taking care of it. As I've stated in the CR it looks good to me, but I would like an explicit approval from Search and some support in testing it after releasing it to make sure it all works fine.

@RKemper @brouberol could you please review this patch, and maybe run a test after it's merged?

@RKemper can I let you do it? You're much more familiar with both ES and our ES/Spicerack operations than I am I think.

Hey @fnegri ! I posted a comment on the CR but just repeating it here.

Unfortunately, the blast radius is pretty big for us...we could lose the ability to do routine operations if the library doesn't work. CCing our software engineers @dcausse and @EBernhardson for awareness.

@Volans I sent you an invite next week to pair on this. Hopefully we should be able to figure this out before our offsite (starting 11 March).

In T345337#9592621, @bking wrote:

@Volans I sent you an invite next week to pair on this. Hopefully we should be able to figure this out before our offsite (starting 11 March).

@bking what if we release spicerack with the change and upgrade it only cumin2002 for few days to allow your team to test and prepare any changes that might be needed to the related cookbooks? Would that be ok?

@bking thanks for having a look! No rush really, I was keen to get this unstuck but it's not a problem if it's not merged until after the offsite.

@bking what if we release spicerack with the change and upgrade it only cumin2002 for few days to allow your team to test and prepare any changes that might be needed to the related cookbooks? Would that be ok?

Sure, that works. Hit me up here or in IRC once it's upgraded and we'll test the cookbooks.

@fnegri no problem, thanks for keeping us in the loop! We'll do our best to get this tested when we can.

Unfortunately, the opensearch library already has at least one breaking change , and it's likely there will be more breaking changes to the opensearch client library. We think it would be better to break off a different library instead of changing "elasticsearch_cluster.py".

That being said, we may be able to fix this error by migrating to a newer version of the curator library. We'll give that a try and update this task.

We're going to upgrade curator (as well as its library) soon, as it's causing other problems (see T360697 ).

In T345337#9652068, @bking wrote:

We're going to upgrade curator (as well as its library) soon, as it's causing other problems (see T345337 ).

The linked task is this same one. Did you meant to link another one?
What do you mean exactly by upgrading curator?

The linked task is this same one. Did you meant to link another one?

Yes, sorry about that...the correct task is T360697 . I've fixed it in my original comment.

What do you mean exactly by upgrading curator?

Per jbond's original comment, "The current dependency on 5.4.1 seems to be comming from elasticsearch-curator" . Curator is an elasticsearch utility that manages index lifecycles. As shown in the linked task, the version we're using is really old, and in fact does not work with our current version of Elasticsearch.

So the idea is to use a newer version of Curator from the 7.x branch. Using the new library should (hopefully) allow spicerack to use newer versions of PyYAML. Feel free to hit me up here or in IRC if you have any other questions about this.

Will you take care also of debian packaging it and any required dependencies?
Because spicerack is deployed with debian packages and upstream debian has 5.8.1 as the most recent release (not sure if has anything to do with the licencing) and 7.17.6 for python3-elasticsearch.

Sure, I'm happy to create a new package. Curator itself is still Apache 2.0 licensed, so I think this is more on us (as in my team) for not recognizing the problem and creating a new package earlier. I'll let you know when a new deb package is ready.

Per subtask, we no longer need to cut a custom package for elasticsearch-curator; the default version provided by Debian should be sufficient. Feel free to remove the pip dependencies for curator and try your build again.

@bking I'm not sure what do you mean. As mentioned earlier in T345337#9658807 Debian has v5.8.1 for python3-elasticsearch-curator and that's the current version used in production. The depedency in setup.py is defined as elasticsearch-curator~=5.0.

Oops, thank you for pointing that out. I'm discussing it further with my team, but we're leaning towards @colewhite 's suggestion here

...reimplement the [curator] calls to change allocation settings using the OpenSearch client directly (client.cluster.put_settings()).

Although in our case, it would be the Elastic client. To that end, I've created T361647 . If you have any other suggestions around implementation, let me know.

Change #1016855 had a related patch set uploaded (by Bking; author: Bking):

[operations/software/spicerack@master] WIP: remove elasticsearch-curator dependency

https://gerrit.wikimedia.org/r/1016855

Change #1016855 merged by jenkins-bot:

[operations/software/spicerack@master] elasticsearch: remove elasticsearch-curator dep

https://gerrit.wikimedia.org/r/1016855

Change #1019686 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/spicerack@master] setup.py: remove dependency elasticsearch-curator

https://gerrit.wikimedia.org/r/1019686

Change #1019686 merged by jenkins-bot:

[operations/software/spicerack@master] setup.py: remove dependency elasticsearch-curator

https://gerrit.wikimedia.org/r/1019686

With the above patch I think the issue should be solved and we can resolve the task. Could some of the original reporters try to repro it again?

Works for me! 🎉

spicerack (master) $ docker run --rm -it -v $(pwd):/src python:3.11 bash

cd /src
pip install .

[...]

Successfully built wikimedia-spicerack python-etcd clustershell

pip install wikimedia-spicerack is also working fine in a Python 3.11 Docker container.