As a Lift Wing user, I want to be able to make one request and return multiple predictions. For example, I want to be able to get revert risk scores for 20 revision IDs using a single API request.
GOAL: This user story is true for a single service.
STRETCH GOAL: This user story is true for one CPU service and one GPU service.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | achou | T348153 Q3 2024 Goal: Lift Wing users can request multiple predictions using a single request. | |||
| Resolved | achou | T348536 Test the kserve batcher for Revert Risk LA isvc | |||
| Resolved | achou | T352987 Implement async revision retrieval and batch prediction for Revert Risk LA | |||
| Declined | achou | T355656 Investigate how to implement batch inference for revertrisk-multilingual | |||
| Resolved | achou | T358744 Deploy RR-language-agnostic batch version to prod | |||
| Resolved | achou | T360406 Error handling in Batch Predictions for RevertRisk Models |
@achou as FYI our Kserve Docker images + deployment charts config should already support the kserve batcher, so you are free to test/enable it anytime :)