The error tracking we have for Mediawiki.org is great, but it doesn't represent the majority of our users. Looking at WebClientError counting most of our errors are occurring on English or Japanese Wikipedia

Is this something @jlinehan can help with?

@Niedzielski yep, does this task have a priority?

@Jdlrobson, @ovasileva can we prioritize this task? We need similar functionality for Vector Vue.js work: T249826.

@JTannerWMF @jlinehan @ovasileva happy to talk through this in a videochat if that's helpful. I think first of all we're just keen to know if it's possible to do this and secondly what needs to happen (and who) to enable this.

@JTannerWMF @ovasileva I think we need to make this high priority and meet with @jlinehan to discuss the problem

To put this in perspective, Our end users are hitting 15.5 million bugs every hour Chrome Mobile and Mobile Safari being the most affected and 15.1 million of those bugs were from Wikipedia. I really think that being able to capture errors from at least one Wikipedia project should help us identify what's happening. For comparison only 2k of errors happen in an hour on mobile on MediaWiki.org

Ideally we'd pick a project from this list: breakdown per project where the risk of enabling it is low and the likelihood of uncovering bugs that affect all our projects is high.

We don't need to distinguish between mobile and desktop. enwiki and jawiki are too big I think.

Looks like this would be covered by T246030, which is about enabling it on a small Wikipedia. The client error instrumentation has since been enabled on haw.wikipedia.org (naturally incl mobile). If that one is too small, perhaps we should enable it on ca.wikipedia (Catalan) instead of haw.wikipedia. Catalan is about 100x larger in traffic (300K views/mo vs 30M view/mo), but still fairly small – mediawiki.org is currently 10M views/mo. (stats.wikimedia.org)

In T255204#6282359, @Jdlrobson wrote:

first of all we're just keen to know if it's possible to do this

Yes it certainly is. Better documentation is coming, for now I'm sorry for it being a little ad-hoc.

and secondly what needs to happen (and who) to enable this.

Background: right now we aren't sampling the errors: if an error happens, it gets sent. There are reasons this might be desirable once the overall error rate comes down (don't want to miss out on catching a rare bug because it was out of sample, etc). The result is that on a high or even medium traffic wiki, the number of error events could be very large and strain some parts of the backend. So the solution for now is to enable the instrument on a low-ish traffic wiki and suppose that the errors there will act sort of like a sample of the errors on any other wiki.

So for what needs to happen, first is to decide on a project that has sufficient traffic that you feel like there is a good enough sample to catch the high-volume bugs. Currently we have it enabled on Hawaiian Wikipedia, but perhaps that is too low-traffic. What do you think about @Krinkle's suggestion of Catalan wiki?

After that, all that needs to happen is a change to the MediaWiki config to enable the instrument on that Wiki. I can help with that.

In T255204#6295586, @Jdlrobson wrote:

@JTannerWMF @ovasileva I think we need to make this high priority and meet with @jlinehan to discuss the problem

I'll reach out to you and set something up so we can talk. I'll put any takeaways back on this ticket.

Catalan could work. According to the break down, Catalan is seeing around 11.9k errors on mobile (from our error counting) every hour compared to 1.6k on mediawiki.org so this seems like should allow us to make some significant discoveries. Thank you!

Per our chat today, @jlinehan and @Mholloway will enable on Catalan Wikipedia. Once that's done please poke me here, and I'll create the required follow up tasks on our side.

@Jdlrobson Thanks for checking, @jlinehan has a separate, more specific task (T258073) which is on the PI kanban board and currently in progress. He advised that this broader task be moved to tracking in the meanwhile. Let me know if you have any concerns!

Thanks! Okay I've repurposed this task as a spike on our end.

I had a look at the new errors coming in and this has exposed T257872 as a bigger problem than I realised. It has high potential for high volume so is certainly is a possibility for the sheer numbers we're seeing on mobile.

@jlinehan after analyzing the logs a little more - the errors we were seeing were actually a large amount of errors from a small amount of users - in fact in a 12hr period, there were 3278 errors on mobile, 3,029 of which came from 1 individual across 2 pages.

Expanding the search query to include extensions and gadgets, I should note that 38,830 of the 48,175 errors logged in a 12 hr period came from the same IP address and same set of user scripts which is insane!

We're going to investigate what effect limiting the counting of errors to a maximum of 5 per page/user session to see what impact that has during next week's deploy on our own error counts but I think what we learn from that is going to be important for rolling out the client side error tracking further - what we're seeing is most errors come from a small amount of places - based on that we may want to one of two things:

• Have some kind of IP blacklist for editors with problematic user scripts/extensions
• Limit the number of errors from a single user/IP like we're doing.

I wanted to cut a bug for that but I wasn't sure where to put it... what do you recommend?

Early signs indicate the error rate is being reduced by almost half since we limited the amount of tracked errors. Am waiting for more data before drawing conclusions.