Page MenuHomePhabricator

PageTriage extension causes TypeError: format.replace is not a function in randomToken function im SearchSatisfaction schema
Open, HighPublic

Description

PageTriage overrides the native Date object in https://test.wikipedia.org/w/extensions/PageTriage/modules/external/date.js?25d10:40 causing bugs in instrumentation of other products.

Seems to impact both Monobook and Vector users
https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-2021.01.25?id=9lwVO3cBjr5R1RLCk-xa
Only occurring on English Wikipedia

at Date.prototype.toString URL1:170:660
at randomToken URL1:108:104
at initialize URL1:110:274
at SessionState URL1:111:367
at setup URL1:119:732
at initialize< URL1:119:976
at atMostOnce/< URL1:118:938
at fn URL1:273:582
at dispatch URL1:277:742
at add/elemData.handle URL1:274:388

URL1: https://en.wikipedia.org/w/load.php?lang=en&modules=ext.centralNotice.choiceData%2Cdisplay%2CgeoIP%2CimpressionDiet%2CkvStore%2CstartUp%7Cext.centralauth.ForeignApi%7Cext.centralauth.centralautologin.clearcookie%7Cext.cite.ux-enhancements%7Cext.cx.eventlogging.campaigns%7Cext.echo.api%2Cinit%7Cext.eventLogging%2CnavigationTiming%2CwikimediaEvents%7Cext.pageTriage.external%2Cinit%2CtoolbarStartup%7Cext.uls.interface%2Cpreferences%2Cwebfonts%7Cjquery%2Coojs%2Coojs-router%2Csite%7Cjquery.client%2Ccookie%2CmakeCollapsible%2CtextSelection%7Cmediawiki.ForeignApi%2CString%2CTitle%2CUri%2Capi%2Cbase%2Ccldr%2Ccookie%2Cexperiments%2CjqueryMsg%2Clanguage%2Cstorage%2Cuser%2Cutil%7Cmediawiki.ForeignApi.core%7Cmediawiki.editfont.styles%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready%7Cmediawiki.page.watch.ajax%7Cmediawiki.ui.button%2Cicon%7Cmmv.bootstrap%2Chead%7Cmmv.bootstrap.autostart%7Cuser.defaults&skin=monobook&version=9ls2e

Replication steps

Expected: Code runs without any problems
Actual: JavaScript error

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Task description:
at Date.prototype.toString URL1:170:660
at randomToken URL1:108:104
at initialize URL1:110:274
function randomToken() {
  return mw.user.generateRandomSessionId() + Date.now().toString( 36 );
}

There is no format variable here, and no method call to anything called replace(). Also, Date#toString is a native method, so from a stack trace perspective it seems impossible for it to be "inside" that method.

I would think that means the error is a "fake". That is, it isn't related to WikimediaEvents in any way, but rather something has messed with built-in prototypes and is just causing random code to fail. Could be a gadget or browser extension, hard to tell.

What is the rate? Any particular browsers?

I do agree it looks a little suspect. 170 errors every 12hrs across 10 IPs

I did wonder if someone was monkeypatching or replacing toString but couldn't find any evidence https://global-search.toolforge.org/?q=Date.prototype.toString+%5C%3D&regex=1&namespaces=&title=.*js

With additional debugging information hopefully some light will get shed.

Change 670010 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] Disable initDebugLogging

https://gerrit.wikimedia.org/r/670010

https://gerrit.wikimedia.org/g/mediawiki/extensions/WikimediaEvents/+/e74af1ec29085f726a07faa0b34bf453d4b00494/modules/ext.wikimediaEvents/searchSatisfaction.js#816

Not sure what's happening here, but it seems to impact a handful of users, and this error is allowing us to identify their browser viewing history (can provide usernames if it helps the investigation).

I'd recommend turning this off anyway, logging twice for an error now we have global error handling seems unnecessary.

Change 670010 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Disable initDebugLogging

https://gerrit.wikimedia.org/r/670010

Jdlrobson claimed this task.

Will check on Thursday to confirm the bug is gone.

Jdlrobson triaged this task as Unbreak Now! priority.
Jdlrobson edited projects, added PageCuration; removed Wikimedia-General-or-Unknown.

I can replicate this now. When I go to https://test.wikipedia.org/wiki/Alberto_jacinto_c._tobias?safemode=1 and type in the input area, I hit the error.
It also applies with safemode=1 https://test.wikipedia.org/wiki/Alberto_jacinto_c._tobias?safemode=1&debug=true

The issue can be traced to https://test.wikipedia.org/w/extensions/PageTriage/modules/external/date.js?25d10:40

This causes 284 errors every 24hrs and impacts instrumentation by modifying a global prototype so while not a train blocker, this should be fixed ASAP.

Jdlrobson renamed this task from TypeError: format.replace is not a function in randomToken function used by SearchSatisfaction schema to PageTriage extension causes TypeError: format.replace is not a function in randomToken function im SearchSatisfaction schema.Tue, Apr 27, 7:07 PM

For me it is https://test.wikipedia.org/wiki/Special:NewPagesFeed where the resource loader module with that script is loaded.

Date.now() returns normally a number, and than the toString is called there

But the linked file has:

Date.now = function() {
    return new Date();
};
...
Date.prototype._toString = Date.prototype.toString;
Date.prototype.toString = function(format) {
    var self = this;
    var p = function p(s) {
        return (s.toString().length == 1) ? "0" + s : s;
    };
    return format ? format.replace(/dd?d?d?|MM?M?M?|yy?y?y?|hh?|HH?|mm?|ss?|tt?|zz?z?/g, function(format) {
        switch (format) {
....

And than Date#toString is called by the randomToken from above. toString is calling the replace function on the given format arg (which is the 36, expected as radix) and that fails.

It is a bad idea to change return types ...

Needs migration to another lib for localization of dates like moment or jquery.ui.datepicker or OOUI elements? Not sure where it is used on that page.

Change 683103 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/PageTriage@master] Drop modifications to Date.prototype

https://gerrit.wikimedia.org/r/683103

I would recommend PageTriage drops that code. The worse thing that can happen from what I can see is that some of the rendered dates will render differently. Looking at the unminified date.js I see a lot of monkey patching and defining of Date.prototype functions.

From what I can see, PageTriage uses this library only for the Date.parseExact function and modified Date.prototype.toString that library provides:

Date.getParseFunction = function (fx) {
    var fn = Date.Grammar.formats(fx);
    return function (s) {
        var r = null;
        try {
            r = fn.call({}, s);
        } catch (e) {
            return null;
        }
        return r[1].length === 0 ? r[0] : null;
    };
};

Date.parseExact = function (s, fx) {
    return Date.getParseFunction(fx)(s);
};

IF we don't feel comfortable deleting, we could unminify the code, and cut out the cruft without the modification of the global Date object.

Example usages:

    parsedTimestamp = Date.parseExact(
					historyItem.get( 'timestamp' ),
					'yyyy-MM-ddTHH:mm:ssZ'
				);

	historyItem.set(
					'timestamp_time',
					parsedTimestamp.toString( mw.msg( 'pagetriage-info-timestamp-time-format' ) )
				);
`

Change 683128 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikimediaEvents@master] Revert "Disable initDebugLogging"

https://gerrit.wikimedia.org/r/683128

Change 683103 abandoned by Jdlrobson:

[mediawiki/extensions/PageTriage@master] Drop modifications to Date.prototype

Reason:

See analysis on ticket. Likely need to find a replacement library or scope this to PageTriage. Can't just remove as the parsing functions this adds seem to be in use.

https://gerrit.wikimedia.org/r/683103

Jdlrobson removed a project: Patch-For-Review.

The last release of DateJs was 2007 and is the one we are shipping in case we were wondering if upgrading was an option :) https://github.com/datejs/Datejs

kostajh added a subscriber: kostajh.

@Jdlrobson is this still considered an UBN?

It depends. I defer to your judgment (with input from search team who's instrumentation is being impacted by this bug).
This error appears to currently be restricted to en.wikipedia.org. Right now I see 131 events in a 12hr period. (Presumably that's because English Wikipedia is the only wiki which has both PageTriage and event logging for SearchSatisfaction setup.)

Per our train guidelines, we block the train when there are over 1000 errors in a 12 hr period for new issues. This is neither an existing issue or over 1000 errors so not UBN by that definition.

Really the priority here comes down to how your team wants to manage their risk and impact on other teams. Given this script is modifying a native browser function Date.prototype.toString, presumably if any new code makes use of Date.prototype.toString then this could hypothetically quite easily lead to a new error with a rate of higher than 1000 errors in a 12 hr period and this will be considered a train blocker. The less users that have page triage rights, the less the risk is here.

Feel free to adjust the priority as you see fit.

Tgr lowered the priority of this task from Unbreak Now! to High.Wed, May 5, 1:51 PM
Tgr added a subscriber: Tgr.

UBN implies a sense of urgency and this bug has been around for a long time so unless it became more disruptive recently e.g. due to improvements to the JS error dashboard, I don't think it's that urgent.