Page MenuHomePhabricator

JavaScript Error on Wikipedia Mobile Sites and Safari: TypeError: $('[accesskey]').updateTooltipAccessKeys is not a function
Closed, DeclinedPublic

Description

Context: There are 220 new errors in the Logstash mw-client-NEW-errors dashboard (beginning on August 3, 2023 to the date this ticket is first published).
Error Information:

  • Error Type: TypeError
  • Error Message: TypeError: $('[accesskey]').updateTooltipAccessKeys is not a function. (In '$('[accesskey]').updateTooltipAccessKeys()', '$('[accesskey]').updateTooltipAccessKeys' is undefined)

Error Context:

  • Domain: Multiple language wikis
  • Version: 1.41.0-wmf.20
  • Skin: minerva and Vector

Actions Required:

  1. Identify the source of the JavaScript error and the reason for the function updateTooltipAccessKeys being undefined.

Additional Information:

  • The logs reveal that the error is specifically occurring in Firefox/Safari browsers in mobile and Safari in Desktop

Event Timeline

There are 220 new errors in the Logstash mw-client-NEW-errors dashboard

Could you please link to the dashboard?

Krinkle subscribed.

@Jdlrobson Can you expand on your hypothesis? E.g. consider the startup module a blackbox that works in the worst way imaginable, how would it cause this error, and why does that seem the most likely explanation?

I've analyzed bugs like this a dozen times before. So far the number of times it was an RL problem is zero. I suggest we assume the same here, as its taking quite a bit of time to investigate on other's behalf.

Can you think of ways to verify your hypothesis? If we suppose that isn't caused by RL, what would be our next step in narrowing down a proximate cause?

Sorry, I meant this looks like an issue in mediawiki.page.ready (mixed up the names)
https://gerrit.wikimedia.org/g/mediawiki/core/+/e5ad274f04780510077f1e8c12a72dee277390eb/resources/src/mediawiki.page.ready/ready.js#81

I've seen this before in a few places where modules that depend on other modules which install jQuery plugins using mw.loader.using throw production errors (not entirely sure why). Previously we had a bunch of errors for $.cookie which went away when we switched mediawiki.cookie to use packageFiles and made $.cookie a wrapper. I think similarly it would make sense for resources/src/mediawiki.page.ready/ready.js to make use of access key via require( 'mediawiki.util' ) rather than jquery plugin.

Is there a better tag for code inside mediawiki.page.ready ? I know there is MediaWiki-User-Interface but it doesn't seem like a great fit (as that's used for a variety of things that don't necessarily mean libraries that live in core). Would it make sense to create for MediaWiki core modules ?

Change 947969 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikimediaEvents@master] clientError: Investigate when mw.util is compromised by third-party script

https://gerrit.wikimedia.org/r/947969

Change 947912 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikimediaEvents@wmf/1.41.0-wmf.20] clientError: Investigate when mw.util is compromised by third-party script

https://gerrit.wikimedia.org/r/947912

Change 947913 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@wmf/1.41.0-wmf.20] mediawiki.util: Investigate when mw.util is compromised by third-party script

https://gerrit.wikimedia.org/r/947913

Aklapper renamed this task from JavaScript Error on Wikipedia Mobile Sites and Safari to JavaScript Error on Wikipedia Mobile Sites and Safari: TypeError: $('[accesskey]').updateTooltipAccessKeys is not a function.Aug 11 2023, 9:35 AM

Likely a low priority given error rate until proven otherwise.

Change 953647 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikimediaEvents@wmf/1.41.0-wmf.24] clientError: Investigate when mw.util is compromised by third-party script

https://gerrit.wikimedia.org/r/953647

Change 953649 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@wmf/1.41.0-wmf.24] mediawiki.util: Investigate when mw.util is compromised by third-party script

https://gerrit.wikimedia.org/r/953649

Change 953649 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.24] mediawiki.util: Investigate when mw.util is compromised by third-party script

https://gerrit.wikimedia.org/r/953649

Mentioned in SAL (#wikimedia-operations) [2023-08-30T21:34:08Z] <krinkle@deploy1002> Started scap: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]]

Change 953647 merged by Krinkle:

[mediawiki/extensions/WikimediaEvents@wmf/1.41.0-wmf.24] clientError: Investigate when mw.util is compromised by third-party script

https://gerrit.wikimedia.org/r/953647

Mentioned in SAL (#wikimedia-operations) [2023-08-30T21:55:26Z] <krinkle@deploy1002> krinkle: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-08-30T22:09:17Z] <krinkle@deploy1002> Finished scap: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] (duration: 35m 08s)

Results are slowly trickling in. As I suspected, some of these appear to be caused by unknown code deliberately re-defining or un-defining properties from other modules after they succesfully load.

Sample 1 at 2023-08-31T20:11:50.862Z

TypeError: $(...).updateTooltipAccessKeys is not a function

  • url: https://en.wikipedia.org/wiki/Special:Contributions
  • user-agent: … Windows … Edg/116 …
  • version: 1.41.0-wmf.24
  • skin: vector-2022
  • gadgets: ReferenceTooltips,formWizard,geonotice,watchlist-notice,WatchlistBase,WatchlistGreenIndicators,SubtleUpdatemarker,charinsert,extra-toolbar-buttons,switcher
  • is_logged_in: true
  • is_mw_util_ready: true true false

I've checked each of the gadgets and those don't seem at fault. The only gadgets in common with the 10 samples I looked at, are those enabled by default, so there'd have to be wider impact on that (unless triggered by a rare interaction with one of the gadgets).

But, that's not all. While I didn't intend for my instrumentation to be able to catch this, we actually see that some of the errors log the value is_mw_util_ready: true true true.

Sample 2 at 2023-08-31T21:53:37.928Z

TypeError: $duplicateLink.removeAttr(...).updateTooltipAccessKeys is not a function

Remember the three values break down as follows:

  1. Bottom of the mediawiki.util module when it is being loaded.
  2. Top of the WikimediaEvents/clientError.js file when it is being loaded.
  3. Callback within clientError.js when the error is being logged.

How can this error happen, with all three values being true? Well, in a synchronous world that would be impossible. After all, if the error happens, it must be undefined, and it "coming back" within the error logging code is easily ruled out. But, it's asynchronous. The above code comes from https://ru.wikipedia.org/wiki/MediaWiki:Vector.js and is fairly easy to fix:

  • site loads, mediawiki.util may not have been loaded yet (as expected, if no dependency is expressed by the caller). The uncaught error is pushed into the mw.track buffer.
  • WikimediaEvents arrives, which awaits mw.util for its own reasons, subscribes to the topic and receives the buffer.

I've fixed the bug for ru.wikipedia.org (edit).

The current rate is about 50-70 per 24 hours. Given no sampling and literally millions of page views per hour, no steps to reproduce it, and our own code ruled out as cause, this is entirely expected noise at our scale. Noting that his is not a backend error where we control the full execution. These are coming from end-user browsers where browser extensions, Greasemonkey, user scripts, and all sorts of self-induced errors can be triggered by accident or on purpose.