As an engineer,
I would like to have a process that tests the deployed model servers after each change in code in order to verify that the models work as expected in production.
This could be some basic testing on the endpoints (something like a smoke test) which at the moment this can only be done manually and it is a burdensome task since we have over 100 models deployed.
A rollout plan for these tests could be the following:
- Identify the inputs for each model and write a script(s) that verifies that we get a 200 response from the endpoints.
- The scripts are run manually from the engineer after deployment
- In the future, once we verify that these tests work well and are sufficient we can enrich them and add them to our CD process.