KServe provides the ability to attach an Explainer to an Inference Service in order to provide an explanation for a prediction given by an ML model. The explanation can be invoked using the :explain endpoint.
KServe integrates with the Alibi Explainer and the AI Explainability 360 (AIX360) toolkit.
Related links:
https://github.com/kserve/kserve/tree/master/python/aixexplainer
https://github.com/kserve/kserve/tree/master/python/alibiexplainer
https://github.com/kserve/kserve/tree/master/docs/samples/explanation