We have some basic guidelines in https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/KServe#KServe_architecture about how to write async code in KServe. We should have a better and more organized set of guidelines, including examples and useful reads. The goal is to point people to this guide as much as possible when they start to write a model server, in order to avoid our ML Engineers to repeat themselves over and over while reading code.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T333453 Lift Wing improvements to get out of MVP state | |||
| Resolved | elukey | T324313 Add guidelines about how to write/use asyncio code in KServe |
Event Timeline
Comment Actions
Opened https://github.com/kserve/kserve/discussions/2578 to ask upstream what are the best practices to use when running cpu-bound code on asyncio.
Comment Actions
Upstream never really followed up with us, so I have updated https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/KServe#KServe_architecture with more recent info (FastAPI, Uvicorn, etc..).
We can add later on code/examples/etc.. based on our users' feedback, no point in adding it now since it may get stale really easily. The overall guidelines seem good enough for the moment!