I added alerts for the user journey tests on enwiki for desktop and emulated mobile as a first start(here, scroll down to "Console errors"). And added dashboards for group 0 and group 1 first view tests but alerts yet, I want to test it some more.

I need to spend some time and verify that we pickup the actually errors in the report before we navigate to a new page. I implemented getting the errors many years ago and I need to verify that through a user journey we pickup every error that comes though the console (and not only the ones that happens when we start to measure and stop to measure the performance). I'll verify how it works tomorrow.

Chrome handles the logs like this: When you start a new browser the logs are empty and then it continues to collect them until you ask to get them. That means that if we do a user journey, we will get all the logs that has been created since start. That means we will not miss any errors and will be safe with the current checks we have.

I created alerts now for the login user journey, mediawiki.org and it.wikipedia.org (with desktop and mobile). The alerts lives in https://grafana.wikimedia.org/d/000000318/performance-synthetic-direct-test-alerts . I tried with console alerts on WebPageReplay tests but we inject some JavaScript that potentially can create errors in the log (I've seen it in a couple of URLS) so its safer to add the logs to the direct tests.