In order to determine whether and when to auto-suspend or remove a mentor, we want to assess whether mentees are being well-served, in particular, whether their questions are being answered and how long it is taking to get an answer.
Proposal: provide a database report showing statistics about mentee questions and the delay involved in answering them. Ideally, I'd like to see a summary report with totals and averages, followed by various detail reports activated by dropdowns or fill-in fields, perhaps a bit like the page views tool. One detail report should be a sortable, by-mentor list showing delay stats (avg, SD, etc.) ideally along with time-series graphs so we can spot trends. I can go into further detail if there is interest.
Note that as a secondary benefit, having these reports available will enable
willing mentors (and other volunteers) to temporarily fill in answering questions that are lagging too long before they are re-assigned. Currently, such unanswered questions are not easy to find.
I can envision a q&d method to at least provide a window into the issue: alter the procedure that places mentee questions on mentor talk pages to include parser conditionals (or a template that I can provide) that, after delay X, categorizes the mentor UTP into Category:Mentor pages with mentee questions unanswered longer than X, and define various delay milestones of interest; perhaps: three days, one week, ten days.
This raises this related question: when you suspend an inactive mentor, what happens to unanswered questions already on their page before suspension?