From @Jdlrobson in T215711:
consider during the render process of a banner catching any errors and hitting a stats beacon with the banner name similar to this:
https://github.com/wikimedia/mediawiki-skins-MinervaNeue/blob/master/resources/skins.minerva.scripts/errorLogging.jsBasically we can catch any errors and increment a counter at wikimedia.event.BannerError.<skin>.<bannerName> to help us identify banners that have problems
In addition to a beacon to monitor live errors, we could add something to banner preview that shows loud JS-error alerts (like a dialogue box, similar to what we have for CSP).