Are you sure it's flaky? It seems to be failing deterministically, and the timestamp length is not an assertion I would expect to have much randomness.

In T299713#7639571, @Tgr wrote:

Are you sure it's flaky?

no, at first I just thought that I have seen this error as a flaky one before.

It’s a MediaWiki core test, but it seems to work on its own (including on my local wiki, where I don’t have PageTriage installed yet). Strange.

I think I found a suspect: https://github.com/wikimedia/mediawiki-extensions-Echo/blob/ceda28121fc9cc4e5ee40e26c6d649e88264ce70/tests/qunit/model/test_mw.echo.dm.NotificationItem.js#L31-L35

	QUnit.module( 'ext.echo.dm - mw.echo.dm.NotificationItem', QUnit.newMwEnvironment( {
		setup: function () {
			this.sandbox.useFakeTimers( now );
		}
	} ) );

Echo’s CI uses fake timers at some point, but does not (as far as I can tell) reset them; PageTriage pulls in Echo (zuul/parameter_functions.py, dependencies).

Change 755961 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Echo@master] Reset test fake timers on teardown

https://gerrit.wikimedia.org/r/755961

Change 755962 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/PageTriage@master] DNM: Test CI

https://gerrit.wikimedia.org/r/755962

Nope, that wasn’t it, https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-noselenium-docker/130792/console still failed.

Change 755962 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/PageTriage@master] DNM: Test CI

Reason:

https://gerrit.wikimedia.org/r/755962

Change 755961 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Echo@master] Reset test fake timers on teardown

Reason:

https://gerrit.wikimedia.org/r/755961

Change 755964 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/core@master] DNM: Add more details to mw.now test

https://gerrit.wikimedia.org/r/755964

Change 755965 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/PageTriage@master] DNM: Test CI

https://gerrit.wikimedia.org/r/755965

Ok, https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-noselenium-docker/130793/console gives us more details:

Match size of current timestamp (1642766492252.74, rounded to 1642766492253, should look like Fri Jan 21 2022 12:01:32 GMT+0000 (Coordinated Universal Time))

Date.now() (MDN) is somehow returning a human-readable date string rather than a number of seconds. I’m guessing some test must be overriding it, though I haven’t found the culprit with codesearch yet.

Wait, lol, this is actually in PageTriage itself?!

https://gerrit.wikimedia.org/g/mediawiki/extensions/PageTriage/+/c9892e8830afd18d6d320a786c88c7f63a049289/modules/external/date.js#41

Date.now=function(){return new Date();};

I assume that’s accidentally missing a + sign, all the polyfills I’ve seen in codesearch were like return +new Date(). Returning new Date() without any type coercion isn’t ideal.

On closer observation, no, it’s not accidentally missing anything, the rest of this external/date.js library actually expects Date.now() to return an object, calling methods like .clearTime() and .add(c) on it. I guess this library (copyright 2006-7) just predates any standardized Date.now(), and decided to add its own custom method to Date.now(), which now conflicts with the standard method that the MediaWiki core test relies on.

I have no idea why this only started failing recently-ish, CI was apparently still green in October.

Suggestions to unblock https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PageTriage/+/755802 (which will resolve two wmf.19 train blockers, so it’s kind of important):

• force-merge
• make the MediaWiki core test use +new Date() instead of Date.now() (could be reverted after the PageTriage change successfully merges)
• get rid of date.js

I don’t think the third option is realistic in time for wmf.19, though it’s certainly the best solution in the long run; any preferences regarding force-merging vs tweaking the core test?

I can force-merge it later once there is no patch in CI (during weekend and quiet time) to avoid too much disruptions.

In T299713#7640030, @Lucas_Werkmeister_WMDE wrote:

Suggestions to unblock […]:

• force-merge
• make the MediaWiki core test use +new Date() instead of Date.now() (could be reverted after the PageTriage change successfully merges)
• get rid of date.js

Which of these options answers the question - Why did CI break now?

That’s still a mystery to me – I don’t plan to investigate that question further, though.

In T299713#7640624, @Ladsgroup wrote:

I can force-merge it later once there is no patch in CI (during weekend and quiet time) to avoid too much disruptions.

Done on master

In T299713#7640678, @Krinkle wrote:

In T299713#7640030, @Lucas_Werkmeister_WMDE wrote:

Suggestions to unblock […]:

• force-merge
• make the MediaWiki core test use +new Date() instead of Date.now() (could be reverted after the PageTriage change successfully merges)
• get rid of date.js

Which of these options answers the question - Why did CI break now?

Maybe T250932: Date.js external include breaks Date.now() (line 41) for modern javascript or T268513: PageTriage's external Date.js modifies/overwrites native JavaScript Date prototype, should be updated trigger this

Since the branch is not cut yet. I think we are out of the woods wrt the train but PageTriage being broken is something that needs fixing.

In T299713#7639936, @Lucas_Werkmeister_WMDE wrote:

I think I found a suspect: https://github.com/wikimedia/mediawiki-extensions-Echo/blob/ceda28121fc9cc4e5ee40e26c6d649e88264ce70/tests/qunit/model/test_mw.echo.dm.NotificationItem.js#L31-L35

FWIW anything that uses this.sandbox gets automatically cleaned up at the end of the test.

Change 755964 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/core@master] DNM: Add more details to mw.now test

Reason:

https://gerrit.wikimedia.org/r/755964

Change 755965 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/PageTriage@master] DNM: Test CI

Reason:

https://gerrit.wikimedia.org/r/755965

Change 730702 had a related patch set uploaded (by SD0001; author: SD0001):

[mediawiki/extensions/PageTriage@master] Replace deprecated date.js library with moment

https://gerrit.wikimedia.org/r/730702

Change 730702 merged by jenkins-bot:

[mediawiki/extensions/PageTriage@master] Replace deprecated date.js library with moment

https://gerrit.wikimedia.org/r/730702