As we begin discussing AI/ML governance and auditing our models, we need to take full inventory of all models currently in production.
Previously, there was the ores-support-checklist, although it has not been updated in ~1 year and is outdated.
At the very least we need the model names, model type, model class and when they were last trained and deployed.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T276397 Model Reporting | |||
| Resolved | • ACraze | T275709 Model Inventory |
Here is a csv file with the current ores model info:
Basic summary:
110 models in production
Largest class: editquality
Largest type/method: gradient boosting
Last trained avg: June/July 2020
Last deployed avg: Oct 26 2020
https://docs.google.com/spreadsheets/d/1s7IUgaBY-VEZzy4xBsnuQgpIfydWVofK3nAj6HqBtv0/edit?usp=sharing
@Chtnnh we are beginning to support other models that are non-ORES-based and don't exactly fit into the ores-support-checklist. We also want to explore the idea of providing updated context around each model to increase transparency and lower the barriers of participation for newcomers and non-experts alike.
I really like the idea of getting automation in place with the new infrastructure, as it reduces the chance of human error and documentation rot. We have an open parent task that discusses how we might automate our model reporting process: T276397
Also, see T276398 for some work on a potential idea around creating on-wiki model cards
Right that makes sense @ACraze!
Do you suggest a complete replacement of ores-support-checklist? I think we can approach this with a combined approach. Like ORES is going into maintenance mode, so can ores-support-checklist, while we develop an alternate that can link back to ores-support-checklist and also includes the non-ORES models.
I have checked out the tasks you mentioned and can see that the general direction is towards automating this task. I think I can back that up and build a component within the new infrastructure that achieves the same. Sounds to me like a good GSoC proposal as well.
What do you suggest I look into?
Marking this a done for now. We can revisit when we are ready to build out a public model registry.
@Chtnnh - yes you have the right idea with the combined approach. I think the next step should be looking into ways to build out automated model cards and eventually a public model registry. Let me know if you have questions about T276398