Absence of a URI doesn't have anything to do with wiki modules specifically. It applies to all modules once they are cached. If anything, wiki modules are cached less often due to them sometimes being user-specific or uncacheable for other reasons. I believe this is why we are seeing a disproportionate amount of errors from gadgets (and much fewer from our own deployed code), exactly because this issue is not affecting wiki modules such as gadgets as much.

While local storage execution has no URL (browser UIs sometimes call this "VM:123" or "(anonymous)" but it seems file_uri remains empty since these aren't valid URIs), it does have a stack trace.

I think it would make sense to simply remove this check, and instead check only for absence of a stack trace, which I suspect was the intent of the check intially - on the assumption that file_uri would be the first uri in any multi-line stack trace. However, we now know that this is incorrect.

In T262493#6586648, @Jdlrobson wrote:

In T262493#6586537, @Krinkle wrote:

leaving aside the known issue that almost all client errors are currently being dropped due to the faulty file_uri check.

That's not correct. As I've pointed out in my investigations in T266517 it looks like we're only dropping client errors in Chrome from ResourceLoaderWiki modules […] I encourage you to do your own investigations here. I am sure you will reach the same conclusion.

Some evidence that goes against your theory:

I already did. I wouldn't have said this based on a theory.

• Right now our highest volume error is in TimedMediaHandler (T262947) […] That was the highest error before we added the file_uri check and continued to be the highest after and wasn't impacted by the change to WikimediaEvents.

When following T262947's task description steps in a fresh Chrome browser with no previous Wikipedia page views, we find an error with a long load.php URL for many freshly fetched modules. When reproducing it again after a refresh, or more generallly on non-initial page views, we still find a load.php URL associated with the error, albeit a much shorter one. This time it fetched only jquery.ui and Kaltura. It having such a URL means it didn't load from localStorage cache.

This is because TMH uses jquery.ui, which is larger than our 100 kB threshold for modules cached in localStorage. As such, they always load from a URL. (From the third page view on, these will enjoy fragmented client-side HTTP-based caching instead).

• If we consider all errors, in a 24hr period 11,771 of our errors come from Chrome 86 and 5,241 from Firefox 82. If I filter by the (albeit flawed) gadgets DSL filter, the Firefox to Chrome ratio changes drastically. (you must remember this has been my obsession for the last month and I've been logging at logstash daily and I would have noticed if we were dropping high profile errors with the file_uri change)

OK. Let's take a closer look. Upon loading the mw-client-errors Logstash dashboard (without any filters enabled) the top five user agents are:

Firefox being more common than any of these major browsers, whilst saddening to my heart, should be suspicious no matter what. Yes, combining the two Chrome patch releases would, just about, give Chrome the top position as most common UA. Having latest Firefox be 80% of Chrome's share is still highly suspect though, and that's before we even filter out unrelated noise from Chrome extensions.

When focussing on the top Chrome version only, I find from spot-checking the top 10 most common error messages that the majority falls in one of two categories:

1. Having a URL like chrome-extension://, which are not related to this task or indeed our sites in general.
2. Having a message like Error: module already implemented, which, as all JS stacks, have their error URL set to where the error is thrown from, not where the stack started. This means these errors have their URL is attributed to load.php?module=startup, the only unfragmented always-cachable load.php URL, which bootstraps the client and thus can't come from localStorage. If we look at any of the individual samples, the stack trace is generally missing the URLs higher up the stack, because the code came from localStorage, attributing it to "", "anonymous", "VM", or (article URL) depending on which browser and how it was loaded. Except for those loaded by users by raw URL (as opposed to by normally loading a gadget module), since that bypasses caching.

All others are from page views where the faulty code was either loaded for the first time on that page and thus not yet cached, or from modules like TMH that are too big for localStorage.

I think it would make sense to simply remove this check

I don't doubt we are dropping events, but right now I am not convinced that we are dropping them significantly. Flooding logstash with errors that cannot be easily debugged due to anonymous stack traces and absent file uris would only give us inflated numbers and more noise to contend with, not alert us to new unknown errors. As the main person championing this it would also cause me unnecessary work.

When the change ( (I9a5fc15fe0aa93a856ab0d835882300f00984d1b ) went live on 19th August, we were still at low volume < 1000 errors a day, adding new wikis and fixing various gadgets as well as hot fixing production code, so it's hard to know exactly how many errors are not being unreported but I should note that deploy didn't drastically drop our numbers:

It's possible we are underreporting the number of errors, but that's okay. Once we have deployed to English Wikipedia, fixed more of our existing bugs, and are seeing a reasonable volume of errors with lots of wiggle room I am happy for us to explore modifying or removing the check, but right now this is keeping triaging manageable.

Here's an easy way to demonstrate the issue.

1. In Chrome, load https://en.m.wikipedia.org/wiki/Main_Page, focus the search bar, and type an "x". (This makes us fetch all relevant code from the server)
2. Refresh the page.
3. Run the following in the console:

mw.trackSubscribe('', console.log.bind(console)); // To see global.error entries
mw.config.values = {}; // This will make MobileFrontend search code crash, unfairly

4. Focus the search bar and type an "x". (This time we're loading code from local cache)

Observe the empty url and thus dropped event:

global.error 
{
  "errorMessage": "Uncaught TypeError: Cannot read property 'prefix' of null",
  "url": "",
  "lineNumber": 168,
  "columnNumber": 306,
  "stackTrace": "at a.getApiData  <anonymous>:168:306\nat a.search  <anonymous>:169:272\nat <anonymous>:192:482\n",
  "errorObject": {}
}

This is applicable to all regular production source code.

In T266517#6604763, @Jdlrobson wrote:

I think it would make sense to simply remove this check

I don't doubt we are dropping events, but right now I am not convinced that we are dropping them significantly.

I hope the above removes any remaining doubt.

Flooding logstash with errors that cannot be easily debugged due to anonymous stack traces and absent file uris would only give us inflated numbers and more noise to contend with, not alert us to new unknown errors. […]

Imho file urls aren't particularly relevant or useful when debugging JavaScript. The stacktrace is all you need imho, which we get either way. Once we have source maps working end-to-end, we can consider compressing variable names for a tiny perfwin, but until then our names are generally more than unique enough to figure things out.

Besides, on the URLs we do get today, it only tells you it was one of 100+ mostly unrelated modules.

I don't see how this could only inflate known errors. Afaik there is no meaningful correlation between loading code from cache and exposing an error. If anything, I imagine it's quite common for most code to first be loaded and not fully excercise all code paths on the first view in that feature's interaction. Pretty much all errors that happen after the first view or use of a feature, we are completely missing out on.

In T274557#6857934, @Jdlrobson wrote:

Right now this is blocking us from expanding our pipeline to load errors from Chrome browser that occur on start up (T266517)

If I understand correctly, you are suggesting that the error load we plan for is limited to the ideal scenario where we don't have significant errors. That sounds dangerous to me. If it is true that we couldn't handle this error happening and while opening up Chrome, then we must not under any circumstances open up Chrome. Whether we fix the Graph bug must not factor into infrastructure capacity, especially infrastructure meant to handle unexpected errors.

Given we have enabled to major wikis already, I assume we have somewhere documented an answer to these two questions or questions similar to it:

• What is the order of magnitude influx we can expect if a majority of Grade A page loads encounters one or two errors? (including which wikis the estimate is based on)
• What is the order of magnitude influx EventGate, Kafka, and Logstash have agreed to support for this feature?

Can you point to these and conclude whether we are currently over capacity? (if it weren't for the Chrome bug). Based on that we need to e.g. reduce wikis, introduce sampling, or apply other measures to limit the maximum amount of influx when unexpected errors happen. Or, if it turns out that the planned capacity already accounts for this and the current rate is orders of magnitudes below it (even with the Graph bug), then it's fine to go ahead.

This is not about eventgate capacity, this is about my own capacity :)

Change 634295 had a related patch set uploaded (by Jdlrobson; owner: Jason Linehan):
[mediawiki/extensions/WikimediaEvents@master] clientError: Adds new fields and updates behavior.

https://gerrit.wikimedia.org/r/634295

We're now in a good place to do this, now that we've removed the user generated content based Graph errors and have a good handle on all our existing bugs.

When allowing these into the pipeline, it's important we can debug them via other means than the file url and filter them if necessary so I've folded this into the pending schema changes.

Change 668141 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] Revise file url logging rules

https://gerrit.wikimedia.org/r/668141

Change 668141 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Revise file url logging rules

https://gerrit.wikimedia.org/r/668141

So this is live now, but not really useful. It seems these additional errors are being anomyized and send as "Script error" without a stack trace so they are totally useless for debugging purposes.

On group 1 I'm seeing 350 errors in the last hour without a file uri, all from Chrome, and not a single one of them is actionable as none of them have useful stack traces. In total there are 516 errors in the last hour. 311 of them are the error "Script error."

So my hypothesise in T266517#6604763 is currently holding true:
We were underreporting errors at the trade-off of actionable errors.

You can filter these with file_url=undefined.

What was your goal here @Krinkle ? I ignore script errors in triaging, so this only impacts the overall error count. It would be useful to understand why code in Chrome loaded from localStorage is anonymized as "Script error" and fix that if possible, but we should resolve this ticket and probably create a new bug for that.

This has led to a huge spike in "Script error"

but only a tiny spike in other errors

I recommend we now either explicitly filter out these inactionable errors or restore the support for filtering out errors where file uri is undefined.

I should note the other errors are pretty useless to us. All of them have stack traces such as "at <anonymous>:1:60" so without the file uri there's little we can do about them.

Change 670945 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] Do not log script errors without file uri

https://gerrit.wikimedia.org/r/670945

Change 670879 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.34] Do not log script errors without file uri

https://gerrit.wikimedia.org/r/670879

Change 670945 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Do not log script errors without file uri

https://gerrit.wikimedia.org/r/670945

Change 670879 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.34] Do not log script errors without file uri

https://gerrit.wikimedia.org/r/670879

Mentioned in SAL (#wikimedia-operations) [2021-03-11T22:34:18Z] <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670879|Do not log script errors without file uri (T266517)]] (duration: 01m 07s)

Mentioned in SAL (#wikimedia-operations) [2021-03-11T22:36:41Z] <brennen> train status: 1.36.0-wmf.34 (T274938): T277229 and T266517 related issues hopefully resolved, rolling forward to all wikis

In T266517#6902959, @Jdlrobson wrote:

[…]

So my hypothesise in T266517#6604763 is currently holding true:
We were underreporting errors at the trade-off of actionable errors.

Were you able to reproduce these scripts errors yourself by triggering an error from a cached script? Did we not already confirm weeks ago that errors from scripts executed after local storage cache have a useful message and trace? They only lack a file becuase they don't come from a networked URL.

I may be wrong here, but I don't think the problems introduced by the above commits have anything to do with the task at hand.

The hypothesis I think is obviously false as indicated by T266517#6604780. The current instrumentation is only logging errors triggered by users in the same page load as when those scripts were downloaded. Not only does that drastically reduce the probability of catching in general (since most script execution happens from a cache, thankfully), it also makes it impossible to catch erorrs from interactions that by design can only happen after a past page load that has downloaded the code in question.

Change 670996 had a related patch set uploaded (by Krinkle; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] clientError: Revert back to ignoring errors with undefined file_url

https://gerrit.wikimedia.org/r/670996

Change 670996 abandoned by Jdlrobson:
[mediawiki/extensions/WikimediaEvents@master] clientError: Revert back to ignoring errors with undefined file_url

Reason:
i debugged this morning with Sam. 1.36.0-wmf.34 is in a weird state, master code is fine.

https://gerrit.wikimedia.org/r/670996