I see that this is caused because of the fastapi version installed with kserve which supports only pydantic versions <2.0.0
You can try to install a newer version of fastapi e.g. fastapi==0.109.0 and see if this solved the issue.
To do this you can just insert the fastapi dependency before the kserve one in the requirements.txt

Thank you for the suggestion @isarantopoulos, I tried fastapi==0.109.0 and run into the error below. It looks like kserve 0.11.2 doesn't support it.

ERROR: Cannot install -r revert_risk_model/model_server/revertrisk/requirements.txt (line 3) and fastapi==0.109.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested fastapi==0.109.0
    kserve 0.11.2 depends on fastapi<0.96.0 and >=0.95.0

I see that kserve does not yet support pydantic v2 and there is work in that direction.
This means that we will have to loosen the knowledge_integrity dependency to allow for versions < 2.0.0 to be installed.

I pinged Muniza about the possibility of loosening the knowledge-integrity constraint to allow for pydantic < 2.0.0 and here is her response:

On Slack, Muniza wrote:

... there are breaking code changes between pydantic v1 and v2 so it won't be possible to just loosen constraints. We'd need to downgrade pydantic in KI to v1 which would require some code changes but more importantly pydantic v1 is supposed to be considerably slower than v2 which might impact the latency of the models.

Since knowledge-integrity 0.6.0 now supports pydantic >=2.1.1 and kserve's fastapi currently supports pydantic < 2.0.0, we will need to wait until fastapi adds support for pydantic > 2.0.0 to effectively run the KI 6.0 performance assessment.

Yesterday I had a discussion about this with Muniza. There is a more recent PR to upgrade to new fastapi and support pydantic v2 in kserve, opened just last week.
https://github.com/kserve/kserve/pull/3374
https://github.com/kserve/kserve/issues/3373

It's unclear when these will be completed and merged into kserve. If it doesn't take too long, we can wait. But if it may take a few months, we would need to downgrade pydantic to v1, which would require making some changes to the code and it would be considerably slower than v2.

Maybe we can ask about the estimated timeline for supporting pydantic v2 in the upcoming Kserve community meeting next Wednesday, so that we can decide our next step.

At the moment it would be great if we could do one of the following:

• have the ability to skip/disable pydantic validation through a flag or something similar. I'm not aware if this is possible so we'll have to investigate if its feasible
• add support for v1 and have the ability to switch between these two. Again I'm not aware if it is possible but it stands from a software engineering point of view.

I would advise against rewriting the code to support v1 instead of v2 as Muniza has done such great work which we would have to redo when kserve moves to v2.
Also waiting for the kserve support would leave us blocked for the time being. Kserve release cycles happen every ~3-6 months and that means we would have to wait for at least 5-6 months.

If you think that my suggestions make sense I can help in assessing the above options. Otherwise we still need to coordinate on which approach to follow.

Following a discussion with Ilias, we will keep an eye on the progress of https://github.com/kserve/kserve/pull/3374. Once the PR is merged, we will use the pre-release version for testing.

The PR for pydantic v2 in kserve has been merged! We can use this commit https://github.com/kserve/kserve/commit/426fe21da0612ea6ef4a116b5114270313e02bbb to test the RRLA model-server :)

Following the workflow we use to build LiftWing model-servers, which involves installing pip dependencies listed in the requirements.txt file. I added the above pre-release commit to the RRLA requirements.txt file, and when I ran pip install -r requirements.txt, the error below was thrown:

Collecting kserve@ git+https://github.com/kserve/kserve.git@426fe21da0612ea6ef4a116b5114270313e02bbb
  Cloning https://github.com/kserve/kserve.git (to revision 426fe21da0612ea6ef4a116b5114270313e02bbb) to /tmp/pip-install-uwt31r3m/kserve_7e7029202b4b49449c96d8b0f6a3185d
  Running command git clone -q https://github.com/kserve/kserve.git /tmp/pip-install-uwt31r3m/kserve_7e7029202b4b49449c96d8b0f6a3185d
  Running command git rev-parse -q --verify 'sha^426fe21da0612ea6ef4a116b5114270313e02bbb'
  Running command git fetch -q https://github.com/kserve/kserve.git 426fe21da0612ea6ef4a116b5114270313e02bbb
  Running command git checkout -q 426fe21da0612ea6ef4a116b5114270313e02bbb
    ERROR: Command errored out with exit status 1:
     command: /home/thevenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-uwt31r3m/kserve_7e7029202b4b49449c96d8b0f6a3185d/setup.py'"'"'; __file__='"'"'/tmp/pip-install-uwt31r3m/kserve_7e7029202b4b49449c96d8b0f6a3185d/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-nd559lke
         cwd: /tmp/pip-install-uwt31r3m/kserve_7e7029202b4b49449c96d8b0f6a3185d/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib/python3.9/tokenize.py", line 392, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-uwt31r3m/kserve_7e7029202b4b49449c96d8b0f6a3185d/setup.py'
    ----------------------------------------
WARNING: Discarding git+https://github.com/kserve/kserve.git@426fe21da0612ea6ef4a116b5114270313e02bbb. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement kserve (unavailable)
ERROR: No matching distribution found for kserve (unavailable)

Since both setup.py and wheel files are missing, this kserve pre-release commit cannot be installed directly via pip.

I have looked into ways to manually build and install this kserve commit locally but these may not align with the standard installation process for LiftWing model-servers, potentially leading to compatibility issues and inconsistencies.

Unfortunately, until this commit is incorporated into an official kserve release, we may not be able to seamlessly use it in our workflow and effectively test the RRLA model-server.

This happens because the kserve repository is not a python package and as the error message tells us there is no setup.py file. The python package can be found in the subdirectory python/kserve.
In order to install it you can add this line in the requirements.txt file:
replace this:

kserve==0.11.2

with this

-e "git+https://github.com/kserve/kserve.git@426fe21da0612ea6ef4a116b5114270313e02bbb#egg=kserve&subdirectory=python/kserve"

Note: the quotes are required otherwise the subdirectory won't be used. For more info you can check the pip VCS support documentation(I got the example from there).

Thanks @isarantopoulos, earlier on I was missing the python/kserve subdirectory. After changing:

kserve @ git+https://github.com/kserve/kserve.git@426fe21da0612ea6ef4a116b5114270313e02bbb

to

kserve @ git+https://github.com/kserve/kserve.git@426fe21da0612ea6ef4a116b5114270313e02bbb#egg=kserve&subdirectory=python/kserve

this pre-release commit was able to be installed. I also checked to confirm the recently added fastapi and pydantic:

pip list | grep -E '(fastapi|pydantic)'
fastapi                   0.108.0
pydantic                  2.6.3
pydantic_core             2.16.3

Now that we are able to install both kserve and KI v6 without encountering dependency conflicts from T355742#9485732, I am going to prepare an environment to perform the runtime performance impact of KI v6.

I noticed that in KI v6, pydantic data models were added to the BaseRevision class in the knowledge integrity schema. The get_revision method used in RRLA relies on the Revision class, which inherits from BaseRevision. Since RRLA uses this method in the preprocess step, I have compared the runtime of the preprocess step for two model servers: one running KI v5 (commit: 026c11a7b3bdb6bd16ef8826bc23b782e8c4e8c8) and another running KI v6 (commit: c8de64b8766e10223eabed73dad1bb2ac68c6b03). Below are the results that show the runtimes based on sample inputs that we use in RRLA load tests and test envs in P58692:

In general, for most request payloads, the preprocess runtime for KI v6 is slightly higher than that of KI v5. However, there are some exceptional cases where KI v6 performs better than KI v5.

On average, there is a slight difference in the preprocess runtime, with KI v5 being slightly faster (0.12) than KI v6 (0.13).

@kevinbazira this is very helpful, thank you!

Please correct me if I'm wrong but I assume the preprocessing time for each payload was calculated over multiple requests? If so, are those values in the Preprocess Runtime (s) column average preprocessing times?

@MunizaA, we're happy to hear that the information provided was helpful. For more context, the preprocessing time for each payload was recorded after every request made in both KI v5 and v6 environments. This is why the average value for the Preprocess Runtime (s) column was calculated in the last row of the table.

Change 1010672 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] RRLA: upgrade KI from v5 to v6

https://gerrit.wikimedia.org/r/1010672

Change 1010672 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] RRLA: upgrade KI from v5 to v6

https://gerrit.wikimedia.org/r/1010672

FYI @MunizaA :)

In T360423#9690484, @achou wrote:

The new RRLA model server featuring KI v.0.6 has been deployed to ML-staging. I used wrk to conduct load testing and compare the performance between the old and new versions. The results for the previous version are under P59447, and the results for the new version are under P59464. From these results, it's clear that the new KI version does not affect the performance metrics, such as average latency and RPS.