Presumably a dependency issue, something not loading the jquery.cookie module. Happening about four times a day. It's happening on enwiki Special:CreateAccount; we don't run any code there, and in any case we don't use cookies. The modules parameter is ext.centralNotice.geoIP|ext.centralauth.centralautologin|ext.cx.eventlogging.campaigns|ext.eventLogging,navigationTiming,wikimediaEvents|ext.uls.common,compactlinks,interface,preferences,webfonts|jquery|jquery.client,cookie,textSelection|jquery.uls.data|mediawiki.String,Title,Uri,api,base,cldr,cookie,experiments,jqueryMsg,language,storage,user,util,visibleTimeout|mediawiki.editfont.styles|mediawiki.libs.pluralruleparser|mediawiki.page.ready|mediawiki.ui.button|skins.vector.legacy.js|user.defaults so it could be any number of things but CentralAuth looks the likeliest.

The stack trace is not too useful, all anonymous methods. It suggests something is using mw.user.sessionId() without the appropriate dependency.

One of the log events has normal stack trace (I suppose that comes down to ResourceLoader localStorage cache vs. normal loading) which implicates https://en.wikipedia.org/w/load.php?lang=en&modules=ext.centralNotice.geoIP%7Cext.centralauth.centralautologin%7Cext.cx.eventlogging.campaigns%7Cext.eventLogging%2CnavigationTiming%2CwikimediaEvents%7Cext.uls.common%2Ccompactlinks%2Cinterface%2Cpreferences%2Cwebfonts%7Cjquery%7Cjquery.client%2Ccookie%2CtextSelection%7Cjquery.uls.data%7Cmediawiki.String%2CTitle%2CUri%2Capi%2Cbase%2Ccldr%2Ccookie%2Cexperiments%2CjqueryMsg%2Clanguage%2Cstorage%2Cuser%2Cutil%2CvisibleTimeout%7Cmediawiki.editfont.styles%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready%7Cmediawiki.ui.button%7Cskins.vector.legacy.js%7Cuser.defaults&skin=vector&version=12p7l line 54 column 880.

mw.requestIdleCallback(function(){if(ignoredSkins.indexOf(skin)===-1){sessionId=user.sessionId();if(isInSample(localConfig.readingDepthSamplingRate))...

That's from readingDepth.js.

...which does have the right dependency so I have no idea what's going on here.

Tldr: user scripts/extensions might be removing $.cookie

https://phabricator.wikimedia.org/T275948
https://phabricator.wikimedia.org/T276368

This is happening on Special:CreateAccount where on-wiki code is disabled (though browser extensions of course aren't). And only happening there - if something changed $ I'd expect way more / worse breakage.

Change 743469 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@master] Disable reading depth on special pages

https://gerrit.wikimedia.org/r/743469

This doesn't look like it relates to reading depth to me (it's only happening because of the scale that reading depth runs at). It relates to the user.sessionId function:
https://gerrit.wikimedia.org/g/mediawiki/core/+/39ebbb075d850c07af114b65973368249ffb117e/resources/src/mediawiki.user.js#158

which calls mw.cookie library:

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/39ebbb075d850c07af114b65973368249ffb117e/resources/src/mediawiki.cookie/index.js

(which has the jquery.cookie dependency)

I think it's fine to disable ReadingDepth on special pages, if we want a quick solution that doesn't solve the root problem: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/743469 (ignore me I haven't had my coffee yet. The error is not just on this page.

Do we want a custom error message here to explore this further? Is there any information that would help us understand this error better?

Change 743469 abandoned by Jdlrobson:

[mediawiki/extensions/WikimediaEvents@master] Disable reading depth on special pages

Reason:

Got confused and misunderstood Gergo. Error is actually occurring on all pages so this doesn't help :)

https://gerrit.wikimedia.org/r/743469

Another theory: It could be that there's some URL filtering active on the target machines that filter out any URLs with the phrase .cookie.

We may also want to consider moving to a more modern better maintained alternative e.g. https://github.com/js-cookie/js-cookie

In T296620#7546786, @Jdlrobson wrote:

This doesn't look like it relates to reading depth to me (...) The error is not just on this page.

Right, sorry, I forgot to disable the Growth filter. So ~1000 errors per day and a wide range of browsers and call sites.

That seems too much for exotic browser extension / user script issues, but too little for it to be unconditionally broken.

In T296620#7546825, @Jdlrobson wrote:

Another theory: It could be that there's some URL filtering active on the target machines that filter out any URLs with the phrase .cookie.

AFAIK ResourceLoader does not execute module code if a dependency failed to load, and here $.cookie is called from mw.cookie and the module containing that does declare jquery.cookie as a dependency.

We may also want to consider moving to a more modern better maintained alternative e.g. https://github.com/js-cookie/js-cookie

Indeed; that's T265703: Replace jquery.cookie with js-cookie in MediaWiki.

Change 771414 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/core@master] TypeError: $.cookie is not a function

https://gerrit.wikimedia.org/r/771414

Why we don't understand why this is happening, some quick testing shows me that when this bug does appear VisualEditor breaks and we drop around 600 analytic events:

https://logstash.wikimedia.org/goto/b0719d73526aee3f230c32fb66269c61

The ISP Mega-Net Dominik Drozdz seems to be the biggest perpetrator.

Both analytics events and VisualEditor do not rely on the cookie value, so the above patch should be considered to make sure these features don't break. @Tgr what do you think (pleae answer after sabbatical.. no rush :-))?

Change 771414 merged by jenkins-bot:

[mediawiki/core@master] TypeError: $.cookie is not a function

https://gerrit.wikimedia.org/r/771414

Change 776333 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] mediawiki.cookie: Revert "Wrap $.cookie in try-catch"

https://gerrit.wikimedia.org/r/776333

Change 776333 merged by jenkins-bot:

[mediawiki/core@master] mediawiki.cookie: Revert "Wrap $.cookie in try-catch"

https://gerrit.wikimedia.org/r/776333

In T296620#7546825, @Jdlrobson wrote:

Another theory: It could be that there's some URL filtering active on the target machines that filter out any URLs with the phrase .cookie.

When code fails to be delivered for any reason (through targeted filtering, or more commonly through a spotty network where 1 request is lost), the rest of the chain is skipped. This happens both when later parts are still in-flight, and when later parts have arrived already and wait for a dependency. ResourceLoader, and browsers more generally when loading module imports, skip execution chains and leave other code unharmed if a dependency is missing. This is something you can rely on as fundamental platform behaviour.

Unless the bug report is for an upstream browser, Apache, or PHP; failed delivery is probably not a cause of JS errors.

In T296620#7541221, @Jdlrobson wrote:

Tldr: user scripts/extensions might be removing $.cookie

In T296620#7542123, @Tgr wrote:

This is happening on Special:CreateAccount where on-wiki code is disabled (though browser extensions of course aren't).

Jon is right. The evidence says a defined method became inaccessible. Assuming we don't do this in a MW extension, something else does. We observe it from pages in safemode (no user scripts/gadgets), which leaves the browser (or extensions to the browser, and network middleware trusted by the browser, e.g. enterprise antivirus modifying pages through a root cert).

To aid future Phab searches, below are some similar logged errors that might share a cause (query "cookie" on Logstash mw-client-errors). Numbers from last 24h. ~500/day total.

TypeError: $.cookie is not a function
~350x, from WikimediaEvents/readingDepth/onVisibilitychange, on Chrome.

SecurityError: Failed to read the 'cookie' property from 'Document': Access is denied for this document.
~200x, from WikimediaEvents/readingDepth/requestIdleCallback/sessionId/$.cookie, on Chrome Mobile (Android).

SecurityError: The operation is insecure
~50x, from document.cookie via $.cookie.

TypeError: can't access dead object.
~50x, from WikimediaEvents/readingDepth/onVisibilitychange/$.cookie, on Firefox.

TypeError: $.cookie is not a function. (In '$.cookie(prefix+key)', '$.cookie' is undefined)
~50x, from WikimediaEvents/readingDepth/onVisibilitychange, on Mobile Safari.

TypeError: Cannot read property 'cookie' of undefined
~20x, from requestIdleCallback/sessionId/mw.cookie.get/$.cookie, on Chrome Mobile (Samsung TV).

Ay ISP/UA pair seems to emit only one of these. This allows for multiple root causes. Or the cause may manifest in different ways. Some can definitely be the same cause merely reported differently due to context (e.g. from a child frame; we discard errors where we are the child frame, but the other way is valid; e.g. VE sometimes uses a headless DOM or child frame), or a different JS engine.

We may also want to consider moving to a more modern better maintained alternative e.g. https://github.com/js-cookie/js-cookie

For half of the above, the failure is at document.cookie which I expect to happen via $.cookie, mw.cookie, a new javascript library, and vanilla JS.

For ones that say of undefined, some global variables were removed or replaced. Not one method. A reverse search on an affected ISP/UA pair also yielded:

• TypeError: Cannot read property 'extend' of undefined.
• TypeError: Cannot read property 'Deferred' of undefined.
• TypeError: $ is not a function.

In T296620#7783009, @gerritbot wrote:

[mediawiki/core@master] TypeError: $.cookie is not a function

https://gerrit.wikimedia.org/r/771414

Catching failures we do not understand, and subsequently have the code take no responsibility for the error, seems unwise. If we find logs distracting, perhaps we're better off turning them off. That would be less damaging to our software and the audience experience. We're not just silencing but altering runtime behaviour to accommodate a rare and unexplained phenomenon. This blocks awareness of far more impactful errors (such as, actually forgetting to declare a dependency!) It also cripples CI. This pattern also tends to break progressive enhancement by allowing broken code to continue, likely beyond the point of no return where the interface can't fallback and remains stuck, instead of gracefully failing early and logging an error. (Example below.)

We, unfortunately, have already seen less experienced contributors follow this trend, by providing patches they believe will solve a problem and improve Wikipedia in some way, after incorrectly learning that the solution to a production error about an undefined thing is to start at the immediate call site and propose silencing it, without assessing the impact, e.g. by || null, if (data.foo && data.foo.bar, isset() or try-catch. This generally causes code to execute in untested or unforeseen ways. In all these years, I have maybe once found this to be an appropriate and helpful remedy, even for the short term. Almost always an undefined error indicates a problem with the caller or the underlying reference (e.g. typo, forgotten case, missing dependency, something actually not being defined somewhere that was meant to be).

These are security or infra implications looming around the corner, too. Both server-side by granting access to private information, or client-side by letting code proceed where previously it stopped and was isolated, can then flood servers, or performs edits on behalf of a user from a condition that is intentionally specific and rare and fails gracefully in the common case, which is then "fixed" and forcefully executing on all pages. This is a mistake for which I've reverted a number of on-wiki script fixes; I learned early on not to do that at our scale, after making such mistake once or twice myself.

Are we going to condition or catch every func call and prop access? Only because some beacon says in some rare and unknown set of circumstances it can throw? This pattern in recent months has imho made numerous code paths worse. See also the principle of "good code is easy to delete". These conditions will either stay forever and grow maintainer uncertainty and prevent refactoring. Or get boldly removed in a subsequent refactor as being impossible/unsupported/incomprehensible (my usual strategy; after tracing the source commit and finding no meaningful evidence).

For example, consider change 673360 where some unknown use of video players may've had a native player instead of TMH due to some unknown problem (likely user script failure, possibly an intended but console-noisy outcome), which was "fixed" by letting TMH forcefully carry on, possibly now removing native controls and replacing them with ones that don't work. Or revision 22294539 which changed a user script that for two months added a button to navigate to action=edit (plus some logspam from a half-pasted script), which was "fixed" two months later without the owner's knowledge to now do something else when clicked, which is to invisibly edit the currently viewed page, change a word, and save it without notification to the user. (Except this disaster was prevented by a typo, mv instead of mw.)

In T296620#7783026, @Jdlrobson wrote:

[…] when this bug does appear VisualEditor breaks and we drop around 600 analytic events:

https://logstash.wikimedia.org/goto/b0719d73526aee3f230c32fb66269c61

I found no evidence in these logs that suggest VisualEditor breaks in this edge case. The async nature of EventLogging should ensure such failures are isolated. If not, it would benefit us more widely to address that resilience issue.

This link currently returns 35 events between March 1 and March 16 (day of comment), and 65 for the month overall. Perhaps not the intended link?

analytics events […] do not rely on the cookie value, so the above patch should be considered to make sure these features don't break.

The analytics events use a cookie as part of the sessionId business logic. Does that qualify as "relying"? Our analysts might not want to reclaim this fraction of lost beacons. Unlike more common causes of lost beacons (e.g. networking) these in particular might muddy the dataset with diverging events not based on documented sessionId behaviour or understood subject application context. Or perhaps it's an insignificant number either way. Discussing a change to this, of course assumes we can actually reclaim them and that clients haven't also corrupted other parts of DOM, jQuery, mw.eventLog, navigator.sendBeacon, etc. and that they are indeed otherwise functional browsing contexts.

Perhaps at unsampled ~500/day, 4x below the 2K/day threshold for holding the train, should we decline it? It seems we have no idea what causes it, nor do we know what the resulting experience is like, or whether it even can be improved. E.g. the "globals undefined" case would likely just move the TypeError a few lines down. (As the above on-wiki revision did, for example), though possibly then getting caught in a different filter that we happen to not look behind, with the code still failing nonetheless, and rewarding us with the aforementioned security risks and added blindspots. Not a net-improvement I would say.